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Bell Laboratories Scientist 
Named Nobel Laureate 


Philip W. Anderson, consulting director in the Physical Research 
Division at Bell Labs, Murray Hill, and a Professor at Princeton Uni- 
versity, has been named one of three joint recipients of the 1977 Nobel 
Prize in Physics. | 

Anderson shares the award from the Royal Swedish Academy of Sci- 
ences with John H. Van Vleck of Harvard University and Sir Neville 
Mott of Cambridge University, England. 

According to the Academy, “The three prize winners are theoreticians 
within the field of solid state physics, the branch of physics which lies 
behind current technical developments, particularly in electronics.” 

The Academy cited Anderson for his central role in developing an 
improved understanding of how local magnetic moments can occur in 
metals, and for his explanation of how electrons become localized in 
amorphous materials. 

Describing the general nature of his work, Anderson said, ‘“‘What I’ve 
been trying to do is to understand the properties of matter, given the 
basic laws of atomic physics. Many of the real materials around us are 
disordered, so-called amorphous materials like glass, in which the atoms 
have no regular arrangement. Many of the properties that we understand 
about solids are supposed to be a consequence of that regular arrange- 
ment. So we had no starting point for understanding disordered sol- 
ids. 

The work the Academy cites has been a starting point for under- 
standing how electrons move in irregular solids. It consists of classifying 
two kinds of situations: One in which the electrons can move freely in 
the whole of the material and the other in which the electrons are pinned 
down, or localized, in one particular place. 

Anderson’s work resulted in a better understanding of why certain 
atoms such as iron are magnetic when dissolved in nonmagnetic host 
metals, why other atoms that might be expected to be magnetic are not, 
and why certain amorphous materials (such as glass) do not conduct 
electricity. 

In 1958 Anderson published a paper in which he showed under what 
conditions an electron in a disordered system can either move through 
the system as a whole or be more or less tied to a specific position as a 
localized electron. This paper, according to the Royal Swedish Academy, 
“has become one of the cornerstones in our understanding of, among 
other things, the electronic conductivity of disordered systems.” These 





Philip W. Anderson in his office at Bell Laboratories, Murray Hill, N.J. 


ideas, the Academy said, ‘‘have been experimentally verified and they 
have in this way laid the foundations for important technical develop- 
ments.” 

Explaining further, Anderson said, “One example that particularly 
interests me is window glass; everyone knows that ordinary window glass 
is a good electric insulator. It’s not a metal. It doesn’t conduct electricity 
well and is used for insulators in power lines and things like that. If you 
look at the reasons in standard physics textbooks for why a substance 
like glass is an insulator, you won’t find answers. This is because these 
materials all depend on the irregularity of the structure, and glass is a 
totally irregular structure. You need this concept of localization to un- 
derstand something as simple as window glass being an insulator.” 
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Anderson also was cited by the Academy for his contributions to the 
basic understanding of local magnetic phenomena. One emerging 
practical application of this theoretical work is increasing use of magnetic 
materials in telecommunications systems and commercial computers. 

Explaining the relationship of his work to bubble technology, An- 
derson said, “I was part of the group many years ago that worked in 
magnetism at Bell Laboratories. The Bell Labs group was a codiscoverer 
of the garnets. Before that, I had formulated a theory which explained 
the kind of magnetism we have in the garnets, and certainly that set the 
stage for understanding these materials. There was even a magnetic 
material that was discovered as a consequence of my theory... . My work 
has almost always been to propose the theoretical background for work 
others do in developing technology.” 

Anderson will be the fifth Bell Labs scientist to be awarded the Nobel 
Prize in Physics. In 1937, Clinton C. Davisson shared the award for 
discovery of the wave nature of matter, which was vital to the subsequent 
development of modern physics and its impact on technology from 
atomic energy to the transistor. The Nobel Prize for the transistor was 
awarded in 1956 to John Bardeen, William H. Brattain, and William B. 
Shockley. 
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The Influence of Rain on Design of 11-GHz 
Terrestrial Radio Relay 


By D. C. HOGG, A. J. GIGER, A. C. LONGTON, and E. E. MULLER 
(Manuscript received April 25, 1977) 


Three salient factors governing attenuation of 11-GHz waves prop- 
agating through rain on a terrestrial radio path are discussed: magni- 
tude of attenuation as a function of rain rate, relationship between 
attenuation and path length, and dependence of attenuation on po- 
larization. Background material is given pertinent to the companion 
papers in this issue which develop procedures useful for radio system 
design. 


Il. INTRODUCTION 


Terrestrial radio systems employing the 10.7-11.7 GHz common 
carrier band have been in use for many years. For example, analog TJ 
and TL radio are used for short-haul applications, and TL has served in 
cross-band diversity as protection for a 6-GHz system. However, new 
emphasis is being placed on autonomous 11-GHz systems, wideband 
digital implementations such as 3-ARDS! being attractive in many ap- 
plications. It is therefore meaningful to re-examine the effects of rain 
on the propagation of 11-GHz signals; hop-length limitations imposed 
by rain have impact on the cost of service. Multipath fading, readily 
accommodated by antennas operated in space diversity, is not discussed. 
In the interest of designing reliable systems, our prime intent here and 
in the companion papers of this issue is to determine the limitation on 
hop lengths imposed by rain attenuation for given fading margin and 
annual outage time objectives for systems in the United States. 
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Three major factors are involved in this determination: 


(t) The 11-GHz attenuation statistics must be properly associated 
with the rain rate statistics as determined by measurements at a point. 
There are two reasons for this: (a) It is only by virtue of long-term mea- 
surements of point rain rate, from sources such as the National Climatic 
Center, that a sufficient quantity of data can be obtained to provide 
reliable temporal statistics for calculation of the path attenuation. (b) 
It is only from measurements of point rain rates at numerous locations 
throughout the country, such as those compiled by the National Climatic 
Center, that the rain environment at arbitrary locations where radio 
systems may be installed can be suitably determined. 

(1) The spatial extent of rain associated with a given point rain rate 
must be accounted for; this is necessary, especially in the case of the 
intense showers which produce large attenuation, because the hop length 
may or may not exceed the dimension of the shower. This interdepen- 
dence between storm dimensions and loss has the effect of producing 
a nonlinear relationship between attenuation and hop length, for a given 
point rain rate. 

(iti) Dependence of rain attenuation on the polarization of 11-GHz 
transmission must be understood. Vertically polarized waves are at- 
tenuated less than those which are horizontally polarized because of the 
oblate shape of large raindrops; this results in different outage durations 
for vertical and horizontal polarization on a hop. In the design of a system 
involving many hops, a sequencing of polarization may therefore be 
desirable to equalize annual channel outage times. 

The above three factors are dealt with in detail in companion papers;” 
here, the background material that forms the basis for these investiga- 
tions is discussed. 


ll. MEASUREMENTS OF ATTENUATION BY RAIN 


In 1956, a transmission experiment* was mounted at 11 GHz on coli- 
near contiguous paths of 20 and 44 km at Mobile, Alabama. The mea- 
sured attenuation caused by rain was compared with attenuation cal- 
culated® from the rain rate measured by 14 gauges along the 44-km path, 
and, although scatter in the data was large, agreement was fairly satis- 
factory. Therefore, by scaling annual distributions of 1-hour point rain 
rates of more than 25 mm/hr, obtained in other regions of the U.S., to 
the Mobile data, a set of contours defining constant hop length for a fixed 
outage time was developed. In 1965, an article® comparing theoretical 
calculations with rain attenuation measured at various microwave 
frequencies in several countries pointed out that serious deficiencies 
existed in the ability to predict path attenuation from point rain 
rates. 
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Fig. 1—Rain attenuation measured on a 1.9-km path in New Jersey compared with 
calculation based upon rain rate measured at a point on the path. 


In 1967, an experiment involving 100 rain gauges arranged in a fine- 
grained network over a 410 km? area was performed at Holmdel, N.J.; 
these data’? showed that the high microwave attenuation events® were 
produced by intense showers of limited size which resulted in nonlinear 
dependence of attenuation on path length, as noted in factor (11) above. 
Simultaneous measurements? at 18 and 31 GHz on a 2.6 km path showed 
that the size distribution of the raindrops in heavy showers is adequately 
represented by the Laws-Parsons size distribution! which has been used 
for theoretical calculation!” of attenuation by uniform rain on a path. 
Direct comparison of attenuation measurements on short paths with 
attenuation calculated from measured point rain rates is fairly satis- 
factory. In the example shown in Fig. 1, a cumulative distribution of 31 
GHz attenuation measured!! during 1968 on a 1.9-km path in New Jersey 
is compared with calculated results based upon point rain rates measured 
near the center of the path. In this early experiment, the predicted values 
of attenuation are somewhat less than the measured values. However, 
more favorable comparisons were obtained later in measurements on 
short paths,!* and are discussed in a companion paper;? these are im- 
portant in accounting for factor (i) of the introduction. 

During that same series of experiments, measurements" at 31 GHz 


INFLUENCE OF RAIN 1577 


DIFFERENTIAL ATTENUATION (A,, — Ay) IN DECIBELS 





10 15 20 25 30 35 40 
11 GHz RAIN ATTENUATION Ay IN DECIBELS 


Fig. 2—Difference between attenuation of horizontally and vertically polarized waves 
measured at 11 GHz on a 23.4-km path in Pennsylvania during 1974. The attenuation on 
abscissa includes wet radome loss. 


showed that horizontal polarization was attenuated about 15 percent 
(in decibels) more than vertical polarization. More recently, measure- 
ments of differential attenuation have been obtained at 11 GHz on a 23 
km path at Harrisburg, Pennsylvania. As shown in Fig. 2, the difference 
amounts to about 12 percent; the effect has been investigated theoreti- 
cally,1516 and good agreement with experimental data is found.!” These 
results are used in companion papers, Refs. 2 and 3, to take factor (iit), 
above, into account. 

In 1970, a substantial experiment!® including 11 GHz propagation was 
started near Atlanta, Georgia. In particular, the 11-GHz rain attenuation 
was measured simultaneously on three hops of quite different length; 
cumulative distributions of path attenuation measured over long periods 
of time could then be compared directly. The comparison was made by 
choosing a (small) time interval of interest to system outage, 30 minutes 
per year for example, and noting the rain attenuations on the three 
distributions at this level of incidence; a certain measured point rain rate 
is also associated with the chosen time interval. These attenuations, when 
plotted versus hop length, are found to increase nonlinearly with dis- 
tance, the slope of the curve decreasing with increasing hop length.? If 
such a plot were to reach an asymptote, attenuation then being constant 
with hop length, the interpretation would be straightforward, namely, 
that well-defined uniform storms with dimensions much smaller than 
the hop length were producing the attenuation. However, this asymptote 
is never reached because the probability of a shower intersecting a path 
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continues to increase as the path length is increased. In reality, rain 
storms are not well defined geometrically, nor is the rain rate uniform 
throughout, therefore calculations!9 involving spatial correlation of the 
rain have been made; comparison with the 11-GHz attenuations mea- 
sured at Atlanta is described in Ref. 3. 


ill. RAIN ON ANTENNAS 


In addition to attenuation on the path per se, rain on various parts of 
an antenna produces enough attenuation to be of concern. This problem 
is especially serious in unprotected paraboloids fed from the prime focus, 
in which case water can form on the aperture of the feed as a layer or as 
drops. Since water introduces both loss and phase shift? at 11 GHz, its 
presence at the feed aperture where the power density of the wave is high 
causes considerable degradation in antenna performance by way of loss, 
reflection, and distortion of the phase of the wavefronts. These antennas 
therefore are usually protected by weather covers which enclose the 
paraboloidal reflector as well as the feed. Thus the problem is relegated 
to transmission through a weather cover wetted by rain; similar con- 
siderations apply to horn-reflector antennas. 

Measurements have been made at 20 GHz on transmission through 
the water layer produced by various rain storms on a radome;?! attenu- 
ations from 4 to 8 dB were observed, depending on factors such as the 
rain rate, wind velocity, and the nature of the surface of the radome. At 
11 GHz, these values scale to between 3 and 6 dB. Measurements at 11 
GHz have been made on a short hop employing paraboloids with hemi- 
spherical weather covers. After the rain on the path was accounted for 
through calculation, each weather cover was found?? to contribute at least 
3 dB of attenuation during heavy rain. Horn-reflector antennas fortu- 
nately are equipped with flat weather covers pitched somewhat beyond 
vertical; recent measurements during rain storms using a pair of closely 
spaced horn reflectors indicate that an attenuation of 4 dB per hop 
should be allowed”? in the fading margin of an 11-GHz system to account 
for water on the weather covers. Note that the attenuation caused by 
water on weather covers is considered part of the fading-margin allow- 
ance rather than part of the attenuation caused by rain on the path per 
se. This allowance is discussed further in Ref. 3. 


IV. SUMMARY 


If the factors discussed above are taken into account, the 11-GHz rain 
attenuation statistics on a hop of arbitrary length can be estimated from 
measured point rain rate statistics as described in the companion papers. 
In all, estimates of annual attenuation have been carried out for 226 
locations in the United States by properly processing point rain rates 
obtained from the National Climatic Center. The occurrence of intense 
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rain is, of course, not the same from year to year at a given location; the 
resulting attenuation behavior will therefore also vary. It has been found 
necessary to use point rain rates measured over periods of 20 years or 
more to generate stable statistics. The deviation of data taken over 1- 
and 5-year periods (from the 20-year value) is discussed in Refs. 2 and 
3. Reference 3 deals with application of the results to radio-path engi- 
neering, and an illustrative example of a tandem-hop situation is given. 
In calculating the performance of a system involving two or more hops, 
it is assumed that no two hops will simultaneously experience very deep 
fading as a result of rain attenuation. This assumption of lack of corre- 
lated outages to rain attenuation leads to slightly pessimistic estimates 
for the outage of a system. 
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11-GHz Radio: 


Nationwide Long-Term Rain Rate Statistics and 
Empirical Calculation of 11-GHz Microwave 
Rain Attenuation 


By S. H. LIN 
(Manuscript received January 27, 1977) 


Two methods are described to obtain long-term (220 years) distri- 
butions of 5-minute point rain rates from data published by the Na- 
tional Climatic Center for U.S. locations. A set of simple empirical 
formulas for converting the distribution of 5-minute rain rates into rain 
attenuation distributions on 11-GHz radio paths has been deduced 
from data measured in Georgia. Additional data measured in several 
other locations also support this empirical formulation. These simple 
formulas and 5-minute point rain rate distributions are useful for path 
engineering of 11-GHz radio. The work on rain rate distributions dis- 
cussed in the paper derives from approaches suggested by the late W. 
Y. S. Chen. 5.9.16 


1. INTRODUCTION 


An important problem in designing terrestrial and earth-satellite radio 
systems at frequencies above 10 GHz is the added path attenuation 
caused by rain. Long term (=20 years) rain rate statistics are needed to 
engineer radio paths for various geographic locations to meet reliability 
objectives. Section II describes a method to obtain 20-year distributions 
of 5-minute point rain rates from the excessive short duration rainfall 
datal.? for locations in the relatively wet eastern and midwestern U.S.A. 
Section III describes another method, employing the theory of extreme 
value statistics, to obtain 50-year distributions of 5-minute point rain 
rates from rainfall-intensity-duration-frequency curves? for locations 
in relatively dry locations such as western U.S.A. Section IV discusses 
the variability of rain rate distributions with observation time base. 
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Table | — Thresholds* of excessive short-duration rainfalls 


Minimum Threshold 
depth of (7 minute 
recorded average 

Duration 7, rainfall, rain rate), 
minutes inches mm/hr 

5 0.25 76.2 

10 0.30 45.7 

15 0.35 35.6 

20 0.40 30.5 

30 0.50 25.4 

45 0.65 22.0 

60 0.80 20.3 

80 1.00 19.1 
100 1.20 18.3 
120 1.40 17.8 
150 1.70 17.3 
180 2.00 16.9 


* Exceeding any one of these 12 thresholds is sufficient to qualify a rainstorm as an 
excessive rainfall. Therefore, this definition does not require that an excessive rainfall 
exceeds all the 12 thresholds. 


In this paper, a “5-minute rain rate” corresponds to the average value 
of the randomly varying rain rate in a 5-minute interval and is calculated 
as AH/r where AH is the 5-minute accumulated depth of rainfall and 
7 = 5 minutes = 142 hour is the rain gauge integration time. Similarly, 
a “r-minute rain rate” is the average rain rate in a 7-minute interval. 

Sections V to VII describe a set of simple empirical formulas deduced 
from experimental data in Georgia for converting the distribution of 
5-minute point rain rates into distributions of rain attenuation on 11- 
GHz radio paths. Section VIII compares the calculated results with 
additional data from other locations. 


ll. 20-YEAR DISTRIBUTIONS OF 5-MINUTE RAIN RATES 


The excessive short duration rainfall data! record details of those 
heavy rainfalls which exceed one or more thresholds; these thresholds 
are dependent upon the rain integration times 7 as shown in Table I. For 
example, the thresholds are 76 and 20 mm/hr for 7 = 5 and 60 minutes, 
respectively. The data, published in tabular form, consist of a storm- 
by-storm compilation of accumulated depth of rainfall in the most in- 
tense 5, 10, 15, 20, 30, 45, 60, 80, 100, 150, and 180 minute periods. For 
example, Table II shows such data for Newark, New Jersey in 1972. Only 
three rainstorms exceeded the excessive rainfall thresholds at Newark 
during 1972. More detailed discussions on the excessive short duration 
rainfall data can be found in Refs. 1, 5 and 6. 

The method for obtaining 5-minute rain rates from this data source 
is illustrated in Table ITI for the storm of August 26, 1972 at Newark. In 
essence, for each storm, the most intense 5-minute accumulation gives 
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Table Il — Excessive short-duration rainfall, year 1972 


Station and 


Date 5 10 1 45 60 80 100 120 150 180 


5 20 30 
et Duration in minutes 


New Jersey 

Newark 
Jul 13 0.26 0.37 0.48 0.57 0.74 0.86 1.06 1.48 1.65 1.80 2.10 2.37 
Jul 17 0.31 0.48 0.49 0.52 0.61 0.70 0.74 0.74 0.74 0.74 0.74 0.74 
Aug 26 0.64 1.08 1.54 1.68 1.80 1.87 1.93 1.98 2.02 2.03 2.05 2.06 


Trenton \ 
. Accumulated depth of rainfall in inches 


Table {Il — Obtaining 5-minute rain rates from excessive short-duration rainfall data, year 1972 
5 10 15 20 30 45 60 80 100 120 150 180 Minutes 
Oe EH. = 195 mm/hr 
5 minutes 


(1.08 — 0.64) inch _ 0.44 inch _ 
(10— 5) minutes 5 minutes 134 mm/hr 


Newark 


Aug 26 0.64 1.08 1.54 1.68 1.80 1.87 1.93 1.98 2.02 2.03 2.05 2.06 
(1.68 — 1.54) inch 


fae 4 h 
(20 — 15) minutes ~ “2 "™™™/F 


one sample of 5-minute rain rate; the difference between the 10-minute 
and 5-minute greatest accumulations gives the second sample of 5- 
minute rain rate; and so on. By applying this single operation to 20 years 
(1953 to 1972) of such data for Newark, New Jersey, we obtain a 20-year 
distribution of 5-minute rain rates in the range above the 76 mm/hr 
threshold as shown in Fig. 1. The results of this method were tested® 
using the reports of the 20 storms which were available in more detailed 
form® and excellent agreement was found.® 


NEWARK, N.J. 
(1953-1972) 


20 YEARS 


AVERAGE NUMBER OF MINUTES PER YEAR 
RAIN RATE EXCEEDS ABSCISSA 





0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 
5-MINUTE RAIN RATE, mm/hr 


Fig. 1—Twenty-year distribution of 5-minute rain rates above the 76-mm/hr threshold 
at Newark, New Jersey. 


As observed in Table I, the published 5-minute rain rates are given 
only for those storms which exceed the thresholds in Table I. These data 
are, therefore, incomplete with respect to 5-minute rain rates of less than 
76 mm/hr. However, in the same publication, progressively lower 
thresholds (see Table I) are used for longer durations; for example, the 
threshold at 60-minute duration is 20 mm/hr. The method developed 
to extend the distributions of 5-minute rain rate to very low rain rate 
employs measured distributions of 1-hour rain rates as the basic data, 
since these are readily available.}-2:7 By using long-term (210 years) data 
from New York City, Miami, and McGill Observatory in Canada, a 
simple empirical formula was deduced for converting the 1-hour rain 
rate distribution into the 5-minute rain rate distribution in the low rain 
rate region.’ A normalizing procedure is used in this conversion formula 
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to account for the difference in rain characteristics between geographic 
areas. T'wenty-year distributions of 5-minute rain rates for locations in 
the eastern and midwestern U.S.A. have been obtained by this process. 
Some results are given in Refs. 8 and 9. 

In low rain rate areas such as Oregon and Washington, however, few 
rainfalls exceed the criterial thresholds, and hence few are included :n 
the excessive short duration rainfall data. For example, at Spokane, 
Washington, the 5-minute rain rate exceeded the 76 mm/hr threshold 
only once in the 20-year interval from 1953 to 1972. In such areas, a 
different data source and a method using statistics of extremes, discussed 
in the next section, is more suitable. 


Ill. 50-YEAR RAIN RATE DISTRIBUTIONS AND EXTREME VALUE 
STATISTICS 

The statistical behavior of the extremes of a random variable has been 
extensively investigated.!°-15 In an unpublished work, W. Y. S. Chen and 
R. L. Lahlum!® have applied the theoretical distribution of yearly 
maximum 5-minute rain rates and an empirical extrapolation to obtain 
the rain rate distribution in the range of interest to radio path engi- 
neering. We extend Chen and Lahlum’s method by incorporating the 
theoretical distributions of the yearly K largest 5-minute rain rates for 
K ranging from 1 to 12. The application of the higher-order statistics of 
extremes eliminates the need for empirical extrapolation. 

Briefly, the average time per year that a rain rate r, measured by a 
gauge with integration time 1, is exceeded, is given by? 





12 K-1e—Ny 
T(Rzr)=w7 > ime > (1) 
K=1 n=0 N! 
for the range T(R = r) < 50 minutes/year, where 
y =a[(Inr) — UJ (2) 


is called the reduced variate, and a and U are scale and location pa- 
rameters, respectively. Notice that T(R = r) in eq. (1) is uniquely de- 
termined by the two parameters a and U. These two parameters can be 
calculated from the rainfall-intensity-duration-frequency curves? which 
are available for U.S. locations. These rainfall-intensity-duration-fre- 
quency curves are derived by the Gumbel method!!-!2 using the theory 
of extreme value statistics and are based on approximately 50 years (1900 
to 1950) of rainfall data. 

From this data source, we need only the following three numbers for 
a given location to calculate the long-term distribution of 5-minute rain 
rates: 
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M = the number of years of rainfall data from which rainfall-inten- 
sity-duration-frequency curves are derived, 

rq = the extreme rain rate with 2-year return period, i.e., the rain rate 
which is exceeded once in 2 years, on average, by the yearly 
maximum 5-minute rain rates, 

ry = the extreme rain rate with 10-year return period, i.e., the rain 
rate which is exceeded once in 10 years, on average, by the yearly 
maximum 5-minute rain rates. 


The formulas for calculating a and U are:9 





a = Anodz v6 (3) 
T 
1 Zo 
U=U.+—|y-—-—= 4 
U Qe | OZ a ( ) 
where 
leo = _ Ag — Ad _ (5) 
Inrg — Inry 
Ag Inrp — Ap Inrg 
ae ee res 6 
reas (6) 
A, = —In (In Oca sem) ~ 0.3665 (7) 
2 years — 1 year 
A, = —In (In Deca tem) ~ 2.95 (8) 
10 years — 1 year 
y = Euler’s Constant ~ 0.5772 (9) 
: | 
ZQ)=-In(- 1 
=n ( ara i ou 
Z=+¥ Z(j) (11) 
M j=1 
Z=—¥ [22 (12) 
M j=1 
and 


Oz —V Z2—Z2? (13) 


For example, Fig. 2 shows a portion* of the rainfall-intensity-dura- 
tion-frequency curves for New York City. The required three numbers 
read from Fig. 2 are 


* The source curves in Ref. 3 cover a wider range for duration 7 from 5 to 1440 minutes 
and return period from 2 to 100 years. 
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2 YEARS 
——— 5 YEARS 
—-— 10 YEARS 


RAINFALL INTENSITY, in./hr 
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DURATION T, MINUTES 


Fig. 2—Rainfall intensity-duration-frequency curve by method of extreme values (after 
Gumbel) for New York City based on 49 years (1903-1951) of rainfall data. 


M = 49 years (1903 to 1951) 
rq = 4.4 inches/hr = 111.8 mm/hr 
ry = 6.5 inches/hr = 165 mm/hr 


By substituting these three values into eqs. (38) to (13), we obtain 
a = 4.363 
U = 4.63 


Substituting this a, U pair into eqs. (1) and (2) yields the 49-year dis- 
tribution (dashed line) of 5-minute rain rates as shown in Fig. 3. The 
49-year (1903 to 1951) distribution obtained agrees well with the 20-year 
(1953 to 1972) distribution obtained by the method based on excessive 
short-duration rainfall data. 

Long-term distributions of 5-minute rain rates for U.S. locations can 
therefore easily be obtained by the extreme value method. 


IV. VARIABILITY OF SHORT-TERM DISTRIBUTIONS OF 5-MINUTE RAIN 
RATES 


Figure 4 shows that the 20-year distributions of 5-minute rain rates 
at Central Park,* La Guardia Airport,* and Newark Airport* agree 


* All within the New York Metropolitan area. 
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Fig. 3—Comparison of 49-year (1903-1951) distribution of 5-minute rain rates calculated 
by extreme statistics method with measured 20-year (1953-1972) data at La Guardia 
Airport, New York City. 


closely. On the other hand, Figs. 5 and 6 show that 4-year (1969 to 1972) 
and 1-year (1972) distributions at these three locations differed signif- 
icantly. Figure 7 displays the convergence of the rain rate distributions 
at Newark as the time base is increased from 1 to 20 years. Figure 8 in- 
dicates that with a single rain gauge measurement, even a 20-year time 
base may still be insufficient to provide stable statistics of extremely high 
rain rates (beyond 160 mm/hr). 

Table IV lists the intervals, in minutes per year, that the rain rate 
exceeded 140 mm/hr at Newark, La Guardia Airport, and Central Park 
during the 20-year observation period. The last column contains the 
three-site summation for each year. The last three rows in Table IV in- 
dicate the 20-year average t, the difference At, between the worst year 
and the best year, and the ratio At/t respectively. Figure 9 shows that 
the normalized range of variations, At/t, increases rapidly as the rain 
rate increases from 80 to 160 mm/hr. This behavior is consistent with 
the divergence between the upper and lower envelopes in Fig. 7. In Fig. 
9, notice that the normalized range of variation, At/t, of individual sites 
are significantly greater than that of three-site summation for high rain 
rates. Since point rain rate statistics may be representative of a short 
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Fig. 4—Twenty-year (1953-1972) distributions of 5-minute rain rates at three locations: 
La Guardia Airport, Central Park, and Newark Airport, in the New York Metropolitan 
area. 


radio hop, but three-site summation statistics are more representative 
for a multihop radio route, then, as shown in Fig. 9, the normalized range 
of variations, At/t, of annual outage time of a radio route can be expected 
to be much smaller than that of a short radio hop. In other words, the 
annual outage time of a radio route is statistically more stable than that 
of ashort radio hop. Intuitively, the statistical stability of the accumu- 
lated outage time of a radio route stems from the partial compensation 
effect of the incoherent, random variations of individual-hop outage 
times as displayed in Figs. 5 and 6. 

The three-site, 20-year measurements yield 60 samples of annual 
accumulated time that rain rates exceed 140 mm/hr as listed in Table 
IV. The average value of these 60 samples is 0.97 minutes per year. Notice 
that 51 out of these 60 samples are less than the average value. Such 
nonsymmetric deviations of small sample data from long-term, large- 
sample average have already been observed and discussed in Ref. 17. 

These data indicate that rain rate statistics gathered from a single rain 
gauge measurement require a very long time base to yield stable statistics 
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Fig. 5—Four-year (1969-1972) distributions of 5-minute rain rates at three locations: 
La Guardia Airport, Central Park, and Newark Airport, in the New York Metropolitan 
area. 
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Fig. 6—One-year (1972) distributions of 5-minute rain rates at three locations: La 
Guardia Airport, Central Park, and Newark Airport, in the New York Metropolitan 
area. 
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Fig. 7—Variations of yearly distributions of 5-minute rain rates at Newark, New Jersey; 
A: 1-year upper envelope, B: 5-year upper envelope, C: 20-year average, D: 5-year lower 
envelope, E: 1-year lower envelope. 
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Fig. 8—Comparison of 19-, 20-, and 21-year distributions of 5-minute rain rates at 
Newark, New Jersey, showing the instability in the extremely high rain rate region. 
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Fig. 9—The effect of three-site summation on the normalized range of variations of 
annual distributions of 5-minute rain rate. At is the difference between the worst year and 
the best year in number of minutes per year rain rate exceeds abscissa during the 20-year 
period (1953-1972), and t is the number of minutes per year rain rate exceeds abscissa 
averaged over the 20-year period (see Table IV). 


Table IV — Number of minutes per year rain rate exceeds 


140 mm/hr 
Three- 
La Guardia Central site 
Time base Newark Airport Park summation 

1953 0.0 0.0 0.0 0.0 

1954 0.0 0.0 0.0 0.0 

1955 0.0 0.0 0.0 0.0 

1956 0.0 0.0 0.0 0.0 

1957 0.0 0.0 0.0 0.0 

1958 0.0 0.0 0.0 0.0 

1959 0.0 0.0 0.0 0.0 

1960 0.0 0.0 5.0 5.0 

1961 5.0 0.0 0.0 5.0 

1962 0.0 0.0 0.0 0.0 

1963 0.0 0.0 0.0 0.0 

1964 0.0 0.0 0.0 0.0 

1965 5.0 0.0 0.0 5.0 

1966 0.0 5.0 0.0 5.0 

1967 0.0 0.0 0.0 0.0 

1968 0.0 0.0 5.0 5.0 

1969 0.0 13.3 0.0 13.3 

1970 5.0 0.0 0.0 5.0 

1971 0.0 6.7 0.0 6.7 

7 1972 8.3 0.0 0.0 8.3 
t (20-year 

average) 12 1.3 0.5 2.9 
At (worst 

year-best 8.3 13.3 5.0 13.3 

year) 
= 6.9 10.2 10.0 4.6 
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Fig. 10—Distributions of 11-GHz rain attenuation measured on three paths near Atlanta, 
Georgia. 


for engineering a radio route. On the other hand, if the time base is not 
sufficiently long, the short term results tend to underestimate the 
long-term, large-sample average. 


V. ANALYSES OF MICROWAVE RAIN ATTENUATION 


For radio path engineering applications, a procedure is needed to 
calculate the rain attenuation distributions on microwave radio paths 
from the available rain rate distributions. Several independent theo- 
retical analyses relating rain attenuation distributions to radio path 
length have been developed.!®?3:30 One analysis is based upon the ap- 
proximate log-normality of long-term distributions of rain attenuation 
and of point rain rates.!7-22 These two distributions are related by 
suitably derived parameters. Since existing theory for converting rain 
rate into rain attenuation applies to spatially inform rain rates, whereas 
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actual rainfalls are almost never uniform over a radio path, the hop is 
divided into incremental volumes in each of which the rain rate is uni- 
form; the total attenuation is then obtained by integrating over the path. 
Since the rain rates at various positions along the radio path only par- 
tially correlated, the increase of attenuation with radio path length is 
nonlinear. 
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Fig. 11—Nonlinear dependence of 11-GHz rain attenuation on path length measured 
at Palmetto, Georgia. 


In another analysis,?° rain cells of circular cross section are assumed, 
allowing calculation of the probable length of path on which rain will fall 
from the probability of rain occurring at a point. It is found that the in- 
crease of attenuation with hop length is nonlinear because of the finite 
diameter of rain cells. In the limit, for hop lengths smaller than the cell 
diameter, the attenuation is almost proportional to path length, but for 
hops much larger than the cell diameter, it is almost independent of path 
length. 

Both analyses indicate that the nonlinearity of the path length de- 
pendence is a function of rain rate and rigorous derivations are fairly 
complex. For path engineering applications, it is desirable to have a 
simple empirical formula to describe this nonlinear behavior. The fol- 
lowing two sections describe the empirical formulas deduced from the 
available 11-GHz rain attenuation data. 
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Fig. 12—Rain rate distributions measured at Atlanta and Palmetto, Georgia. 


Vi. RAIN ATTENUATION AND RAIN RATE DATA AT ATLANTA 


The 11-GHz rain attenuation distributions obtained by simultaneous 
measurement on three paths (5.07, 25.6, and 42.5 km) near Atlanta”4 
from November, 1973 to July, 1974 are shown for illustration in Fig. 10. 
The path-length dependence derived from these data is indicated in Fig. 
11 which is a cross-plot relating attenuation observed to path length 
traversed, for fixed levels of probability. These data demonstrate that, 
for probability levels of 0.05 and 0.1 percent, rain attenuation increases 
nonlinearly with increased path length. 
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Fig. 13—Nonlinear dependence of 11-GHz rain attenuation on path length measured 
at another location. 


Rain rate data for the above measuring interval are available from a 
tipping-bucket rain gauge at a common path terminal (Palmetto, 
Georgia) and from a Weather Bureau rain gauge at Atlanta Airport.!:.3 
Figure 12 shows the distributions of 5-minute point rain rates obtained 
from these two rain gauges, as well as long-term results from the same 
Atlanta station. 

Figure 13 shows another example of nonlinear increase of rain atten- 
uation with path length measured at another location. The data in Figs. 
13 and 15 are from the same city. 


Vil. EMPIRICAL PATH-LENGTH DEPENDENCE 


Many authors2&*5 have pointed out that the relationship between the 
rain attenuation gradient, 8 in dB/km, and the point rain rate, R in 
mm/hr, can be approximately described by 


B(R) = pR” (14) 


where the coefficient p and the exponent 7 depend on the radio fre- 
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Fig. 14—-Comparison of calculated (dashed lines) versus measured 11-GHz rain atten- 
uation distributions on 5.07, 25.6, and 42.5 km paths at Palmetto, Georgia. The calculations 
are based on 20-year distribution of 5-minute point rain rates at Atlanta, Georgia. 


quency, the polarization of the radio signal, and the canting angles of 
the oblate raindrops. Based on T. S. Chu’s theoretical calculation®® and 
the experimental data on the polarization dependence of rain attenuation 
in Refs. 37 and 38, the empirical formula for 11-GHz rain attenuation 
gradient is 


Bvy(R) = 0.0153R1:1999  dB/km (15) 
for vertically polarized signals and 


By(R) = 0.0170R1-2012  dB/km (16) 


for horizontally polarized signals. 

If the rain rates were uniform over a radio path of length L (km), the 
path rain attenuation u(R,L) would be simply 8(R)L, but since actual 
rainfalls are not uniform, the increase of u(R,L) with L is nonlinear. In 
Ref. 39, a two-parameter empirical formula was shown to describe this 
nonlinear behavior. A single parameter variation also provides satis- 
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Fig. 15—Comparison of calculated (dashed lines) versus measured 11-GHz rain atten- 


uation distributions on 11.3, 20.8, and 32.1 km paths at City A. The calculations are based 
on 20-year distribution of 5-minute point rain rates at City A. 


factory agreement: 


w(R,L) = B(R)L —— (17) 
1+ L(R) 
where the factor 
: ; (18) 
1+ LR) 


accounts for the partially correlated rain rate variations along the path, 
and L(R) is a characteristic path length such that the nonlinear factor 
(18) equals one-half when L = L. L is related to the diameter of the rain 
cell. 
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Fig. 16—Comparison of calculated (dashed line) versus measured 11-GHz rain atten- 
uation distribution on a 10.1-km path at City B. The calculation is based on 20-year dis- 
tribution of 5-minute point rain rate at City B. 


By fitting eq. (17) to concurrently measured (August 1973 to July 1974) 
distributions of 5-minute point rain rates and 11-GHz rain attenuation 
on the 42.5 km path at Palmetto, Georgia, it is found that L(R) can be 
approximately described by 


2636 


R-62 km for R>10mm/hr (19) 


L(R) ~ 


Vill. COMPARISON OF PREDICTED AND MEASURED 11-GHZ 
ATTENUATIONS 

Measured rain attenuation data on nine 11-GHz paths, listed in Table 
V, are available for comparison with the calculated results. Figures 14 
to 18 show such comparisons. The calculated results (dashed lines) are 
based on 20-year distributions of 5-minute point rain rates and include 
path rain attenuation and assumed wet radome loss listed in Table V. 
It is assumed that, on the average, a wet, flat radome introduces 2 dB 
loss and a wet, hemispheric radomes introduces 4-dB loss.1®40.4! Figures 
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Fig. 17—Comparison of calculated (dashed line) versus measured 11-GHz rain atten- 
uation distribution on a 12.2-km path at City C. The calculation is based on 20-year dis- 
tribution of 5-minute point rain rates at City C. 
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Fig. 18—Comparison of calculated (dashed line) versus measured 11-GHz rain atten- 


uation distribution on a 32.3-km path at City D. The calculation is based on 20-year dis- 
tribution of 5-minute point rain rates at City D. 
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Location 


Palmetto, Ga. 
Palmetto, Ga. 
Palmetto, Ga. 
City A 
City A 
City A 
City B 
City C 
City D 


Table V — Measured rain attenuation data on 11-GHz paths 


Path 
length, 
km 


42.5 
20.6 

5.1 
32.1 
20.8 
11.3 
10.1 
12.2 
32.2 


Time base 


8/73-7/74 
8/73-7/74 
8/73-7/74 
74-75 
74-75 
74-75 
74-75 
8/72-12/75 
9/75-8/76 


Radomes 


2 flat radomes 

2 flat radomes 

2 dishes without radome 
2 flat radomes 

2 flat radomes 

2 flat radomes 

1 flat; 1 hemispheric 

2 hemispheric radomes 
2 flat radomes 


Polar- 
ization 


mmom<<<2ro< 


Assumed attenuation 
(dB) due to two wet 
radomes for 
calculations 





ROO LAL LP PR 


14 to 18 indicate that the measured results are comparable with the 
predicted curves. 


IX. CONCLUSION 


Two methods have been described to obtain long term (=20 years) 
distributions of 5-minute rain rates from data published by the National 
Climatic Center for U.S. locations. Some typical results are given in Refs. 
8 and 9. The variability of distributions based on shorter terms is dis- 
cussed in Section IV. 

A set of simple empirical formulas, for converting the distributions 
of 5-minute rain rates into the distributions of 11-GHz rain attenuation 
on any path length, has been deduced from experimental data gathered 
near Atlanta, Georgia. These formulas are supported by further exper- 
imental data from other locations. 
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Radio Path Engineering 


By T. L. OSBORNE 
(Manuscript received February 1, 1977) 


This paper describes the procedures adopted at Bell Laboratories 
for using rain attenuation data to engineer 11-GHz microwave radio 
hops and routes. Rain outage time charts, which show the rain outage 
time as a function of rain attenuation and hop length, are the basic tools 
in engineering the hops. The charts, their formulation, and the pro- 
cedures for using the charts are described and illustrated with several 
examples. The procedures are used to demonstrate and quantify the 
sensitivity of allowable hop lengths to the available rain attenuation 
margin, the effects of a limited rain attenuation margin, and the effects 
of the variation in the outage ina single year from the 20-year average 
outage. Guidelines for judging if a hop or route is performing as engi- 
neered are developed. 


|. INTRODUCTION AND SUMMARY 


In engineering a microwave radio system, as in engineering any system, 
one of the major concerns is the amount of time that the system will not 
be usable or that its performance will be below an acceptable level; this 
is known as outage time. For a system to be reliable, the amount of outage 
time must not exceed some objective and should be controllable and 
predictable. 

In microwave radio systems, outages can be separated into two classes 
according to source: failure of the system equipment, and anomalous 
propagation conditions. In modern systems, the equipment outage time 
can be made negligibly small by using standby equipment and automatic 
protection switching systems. Protection against some propagation 
outages can also be achieved by providing an alternate path or channel 
and automatic switching; multipath fading protected by space or fre- 
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quency diversity is an example. However, at frequencies above about 
10 GHz, attenuation by rain can cause an outage which is not easily 
protected by providing an alternate path or channel because rain at- 
tenuation is relatively constant with frequency across the common carrier 
bands, attenuates both polarizations, and covers a fairly large area. The 
use of route diversity to provide an alternate propagation path has been 
considered, but because of the cost of providing a complete standby 
system and the uncertainty of the amount of joint fading on the two 
routes, this has not been practical. 

Therefore, the only practical way of achieving a reliable radio system 
at those frequencies where there is substantial rain attenuation is to 
engineer the system in such a way that the expected amount of rain 
outage is below some outage objective. However, reduction of the amount 
of rain outage is primarily achieved by reducing the radio repeater 
spacing, which in turn means increasing the number of radio repeaters. 
Because radio repeaters are very expensive, this can greatly increase the 
cost of a system. Conversely, in order to reduce the cost of the system 
as much as possible, the system must be engineered for the longest hops, 
and consequently the fewest repeaters, for which the rain outage will 
meet the outage objective. The unavoidable dependence of the eco- 
nomic-versus-reliability trade-off on a statistical occurrence of nature 
is peculiar to radio systems operating at frequencies above about 10 GHz, 
and makes a reliable practical method of path engineering crucial to the 
future use of 11-GHz radio systems. 

Prior to the availability of the data described in these papers, most 
radio paths were engineered using data and methods developed by Ha- 
thaway and Evans and published in 1959.1 While this work did provide 
a methodology, it was based on only six months of data on two hops in 
one city which was then related by not-well-established relationships 
to rain data in other parts of the U.S. By the mid-1960s, as the future use 
of more and higher-capacity 11-GHz radio systems became apparent, 
descrepancies and problems with rain attenuation theory in general were 
pointed out,” and complaints of excessive outage in some existing systems 
were reported, it became obvious that better design information was 
needed. 

Collectively the companion papers in this issue describe the current 
theories for predicting the amount of rain outage, the underlying sub- 
stantiating data, and the methodology for engineering 11-GHz radio 
systems limited by rain attenuation. The paper by Hogg et al.® reviews 
the factors involved in developing the rain attenuation theory and the 
work which led to the present understanding. The paper by Lin‘ de- 
scribes the source and processing of the basic rain rate data, and the al- 
gorithm for converting the rain rate data to rain attenuation data. 

This paper describes the procedures which have been adopted for 
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using the rain attenuation data to engineer 11-GHz microwave radio hops 
and routes. Section II describes the rain outage charts which are used 
for radio path engineering. Section III describes, and illustrates with 
examples, methods of using the rain outage charts to estimate annual 
outage time and allowable hop length to meet a given objective. The 
methods are then used to demonstrate the sensitivity of allowable hop 
lengths to the available rain attenuation margin and the effect of a 
limited rain attenuation margin. Section IV uses the methods of Section 
III to demonstrate the effects of the variation in the outage in a single 
year relative to the 20-year average. Section V discusses the geographical 
coverage of the rain attenuation data and considerations in estimating 
rain outage in areas where no data exist. 

In addition to the methodology just described, some quantative data 
is derived and is summarized as follows. 


(t) The differential attenuation between the horizontal and vertical 
polarization is 8.0 to 8.5 dB for 50 dB of attenuation on the vertical po- 
larization and hop lengths from 60 km to 10 km respectively (Section 
3.2). 

(ii) A 5-dB difference in rain attenuation margin results in a 16 to 
18 percent difference in allowable hop length, and a 10-dB difference 
results in a 30 to 35 percent difference in allowable hop length (Section 
3.4). 

(iii) For midcontinental cities in the U.S., the factors by which the 
maximum 1-year outage exceeds the 20-year average outage range from 
2.5 to 7.1; the variability for the western cities is slightly more. Factors 
are also given for the maximum 5-year averages (Section 4.1). 

(tv) Guidelines are developed for judging if a system rain outage 
performance is as engineered. For example, if the route outage time of 
a route containing three or more hops exceeds the engineered value by 
more than a factor of 5 in any one year, or by a factor of 2 for any 5-year 
average, the outage time is excessive and the reason should be deter- 
mined. (Section 4.1). 

(v) If hops were engineered on the basis of the maximum 5-year 
average outage time, 9 to 27 percent more repeaters would be required 
than if they were engineered on the basis of the 20-year average. From 
25 to 77 percent more repeaters would be required to engineer on the 
basis of the maximum one-year outage time (Section 4.2). 


This paper addresses the problem of outage caused by rain attenuation 
only. In some cases there may be significant amounts of outage due to 
other causes such as multipath fading or equipment failure. In such cases, 
the basic procedure is to allocate only part of the total allowable outage 
to rain attenuation. Detailed considerations of the allocation and com- 
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putation of outage time from other effects is outside the scope of this 
paper. 


ll. RAIN OUTAGE CHARTS 


The generation of rain attenuation statistics as described by Lin‘ is 
a two-step process. First, long-term appropriately averaged, point rain 
rate distributions are derived from weather bureau data at a given 
location. These distributions show the number of minutes per year, 7’, 
that the 5-minute point rain rate, R, exceeds a given value, and can be 
described by the functional relation 


T = g(R) (1) 


Secondly, the radio path attenuation, a, for a given hop length, L, and 
polarization is related to the 5-minute point rain rate, R. From eas. (14) 
through (19) in Lin’s paper,‘ these relations for vertical and horizontal 
polarizations are 








0153R1-19097, 0170R1-20121, 
ee R-62 “ene P26 @) 
PAL ( : ) L+L ( ) 
2636 2636 


where a is in dB, R is in mm/hr, and L is in km. 

In radio path engineering we are interested in the amount of time the 
path attenuation exceeds the rain attenuation margin, Mp, on the radio 
path. This can be determined by setting a equal to Mr and solving eq. 
(2) for the rain rate which gives the marginal attenuation, then using the 
rain rate distribution to find the amount of time that rain rate is ex- 
ceeded. Functionally, the solution of (2) for R can be represented by 


R = fy (Mp, L) (3a) 
R= fx (Mp, L) (3b) 


for the vertical and horizontal polarizations respectively. Figure 1 and 
2 show the rain rate R as a function of rain attenuation margin with hop 
length as a parameter for 12 dB < Mr < 70 dB and 10 km < L < 60 
km. 

For engineering purposes, a rain outage chart should show the annual 
expected rain outage time as a function of the rain attenuation margin 
and hop length for both vertically and horizontally polarized transmis- 
sion at a given location. Such charts have been devised by solving eqs. 
(3) and (1) graphically by juxtaposing the rain rate scales of Figs. 1 and 
2 with the rain rate scale of the point rain rate distribution. Examples 
of the resulting charts are shown in Figs. 3 to 9; Fig. 3 shows the rain rate 
scale for illustration only. 
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V POLARIZATION HOP LENGTH, km 


5-MINUTE POINT RAIN RATE, mm/hr 





10 20 30 40 50 60 70 
RAIN ATTENUATION MARGIN, dB 


Fig. 1—Graph of the empirical relation between the annual distribution of 5-minute 
point rain rates and the annual distribution of rain attenuation of the vertical polarization 
on a radio hop with the hop length as a parameter. 


The advantages of this type of rain outage chart are: (i) the rain rate 
distributions are displayed explicitly; (ii) the graphical solutions of eqs. 
(1) and (3) are kept independent, which simplifies the work required in 
changing the charts should new or revised data become available; (iz) 
the rain rate values are available if needed although not explicitly shown; 
(tv) both horizontal and vertical polarization data is shown on the same 
chart; (v) the rain attenuation margin-hop length scales are the same 
for every chart. 
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H POLARIZATION HOP LENGTH, km 


5-MINUTE POINT RAIN RATE, mm/hr 





10 20 30 40 50 60 70 
RAIN ATTENUATION MARGIN, dB 


Fig. 2—Graph of the empirical relation between the annual distribution of 5-minute 
point rain rates and the annual distribution of rain attenuation of the horizontal polar- 
ization on a radio hop with the hop length as a parameter. 


lll. USE OF RAIN ATTENUATION CHARTS 


3.1 Determination of rain attenuation margin from system parameters 


The first step in using the rain attenuation charts for engineering a 
radio route is to determine the available rain attenuation margin from 
the specifications of the equipment used on each hop. This section de- 
scribes the procedure and gives an example using typical values. 

The basic equipment specification is the system gain, G,, for a given 
performance threshold, which is defined as the dB difference in signal 
levels between the transmitter bay output and the receiver bay input 
for the given performance threshold, where the channel combining 
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Fig. 3—Illustration of the use of a rain outage chart to find the outage time on the vertical 
and horizontal polarizations from a known rain attenuation margin and hop length. 


networks are assumed to be inside the bays. In digital radio systems the 
performance threshold is usually specified in terms of bit error rate 
(BER), whereas in analog radio systems it is specified in terms of voice 
frequency channel noise. The total fade margin, M7, against rain fading 
is the amount of flat signal loss that degrades the system performance 
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Fig. 4—TIllustration of the differential attenuation between the horizontal and vertical 
polarizations for hop lengths from 10 km to 60 km and an attenuation of 50 dB on the 
vertical polarization. 


to a given threshold in the absence of any other degradations. It can be 
found from the system gain for the same threshold by subtracting the 
section loss, L;, which is the sum of the waveguide, antenna system, and 
free-space path losses less the antenna gains. 

The total fade margin must be allocated to the various losses and 
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Fig. 5—Illustration of the use of the system characteristic curve and the system objective 
curves in finding the maximum allowable hop lengths for a particular system. 


degradations that occur during rain fading, such as rain attenuation in 
the aerial path, wet radome loss, depolarization performance degrada- 
tion, and foreign system interference degradation. The rain attenuation 
margin, Mp, is that margin which is allowed for aerial attenuation only, 
and is not necessarily equal to the total fade margin. 

Wet radome losses have been discussed by Hogg et al.? For engineering 
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Fig. 6—IIlustration of the vertical and horizontal polarization outage times for a system 
engineered to meet a weighted average outage time. 





11-GHz radio systems using antennas with flat vertical radomes, a total 
loss of 4 dB for both antennas is normally assumed.° In digital radio 
systems using dual polarized frequency channels, depolarization by 
heavy rain can cause cochannel interference to degrade the system 
performance by about 2 dB.° Foreign system interference can cause a 
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Fig. 7—Illustration of the effect of a rain attenuation margin which is limited by the 
radio receiver AGC range. 


performance degradation during rain fading if the carrier-to-interference 
(C/I) ratio approaches the fade margin plus the system carrier-to-noise 
ratio at the performance threshold, and the interference does not fade 
with the desired signal. Normal frequency coordination practices require 
C/T ratios so high that this effect is negligible. However, if the desired 
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C/I ratio cannot be achieved, then the reduced C/I ratio can be tolerated 
by engineering with a reduced rain attenuation margin. 
Table I shows an example calculation of the rain attenuation margin 
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average, and a maximum 1-year average outage time; and of the maximum hop lengths 
which will meet an objective for the maximum 5-year average and maximum 1-year outage 
times. 


for a typical 11-GHz digital radio system path with a length of 40 km. 
For this example the rain attenuation margin is 50 dB. 

Since the rain attenuation margin for a given type of equipment de- 
pends on the components of the section loss, these components can be 
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Table | — Calculation of rain attenuation margin (example) 


Parameters Decibels 
System gain at 10-3 BER, G, 112.0 
Waveguide loss, total 6.6 
Free-space path loss, 40-km path 145.6 
Antenna gain, total for two 10-foot dish antennas 96.6 
Section loss, L, 55.6 
Total fade margin available for rain fading (M7 = G, ~— L;) 56.4 
Wet radome loss, L, 4.0 
Depolarization performance degradation, Lxypp 2.0 
Foreign system interference degradation, Ly 0.4 
Rain attenuation margin, (Mp = Mr — L, ~ Lxpp — Lr) 50.0 


chosen to give the optimum reliability versus economic tradeoff for each 
path. For example, if a path is constrained to be short because of terrain 
or the need for dropping a channel, then cost can be reduced by using 
less expensive but more lossy waveguide, or smaller antennas. 

However, as is shown in Section 3.4, the allowable path length is quite 
sensitive to the rain attenuation margin. Because radio repeater site costs 
are so large in comparison to waveguide and antenna costs, it is usually 
least expensive to engineer for the longest path possible. 


3.2 Determination of per-hop outage from rain attenuation margin 


Once the rain attenuation margin and hop length are known, the ex- 
pected number of minutes per year the hop performance will be below 
the performance threshold can be read directly from the rain chart. 
Figure 8 illustrates the use of the chart for the system used in the pre- 
vious example. The lower scale is used for the outage on the vertical 
polarization and the upper scale for the outage on the horizontal polar- 
ization. For this example, the vertical polarization outage time, T'y, is 
11 minutes and the horizontal polarization outage time, Ty, is 19 min- 
utes. 

The difference between vertical and horizontal polarization outage 
times is caused by the differential attenuation between the two polar- 
izations. For 50 dB of attenuation on the vertical polarization, the dif- 
ferential attenuation ranges from 8.0 dB to 8.5 dB for hop lengths from 
60 km to 10 km as illustrated in Fig. 4. The differential attenuation is 
relatively insensitive to hop length but varies substantially with absolute 
attenuation as shown in Hogg et al.® 

A short-haul radio annual outage objective of 0.02 percent for a 
250-mile system is often used in the Bell System. This amounts to 105 
minutes per year for a 400-km system, or 10.5 minutes for a 40-km hop. 
Thus the expected annual outage for the vertical polarization is just over 
the objective, while the outage for the horizontal polarization is sub- 
stantially higher than the objective. 

In medium to highly loaded systems, both polarizations must be used 
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and the hops must be engineered to give adequate reliability with both 
polarizations utilized. Because some form of polarization frogging can 
usually be used to average the outage of both polarizations on any one 
trunk, and because it is almost always possible to use the vertical po- 
larization more than the horizontal, the hop outage is usually taken to 
be the weighted average outage, weighted 60 percent for the vertical and 
40 percent for the horizontal polarization. In this case the weighted av- 
erage outage is Tay = 0.6 T'y + 0.4 Ty = 14 minutes, which is still above 
the objective. 

In order to meet the objective the rain attenuation margin must be 
increased. This can be done by changing the section loss or by shortening 
the hop length. From this point on, the procedure is by trial and retrial. 
The next section describes a method for finding the maximum allowable 
hop length by graphical construction. 

The allocation of an objective on a per-hop basis by prorating the route 
objective on the basis of hop length implicitly assumes that the fading 
events on each hop are mutually exclusive. Since some simultaneous 
fading of adjacent or nearly adjacent hops is expected, this procedure 
leads to pessimistic estimates of the total outage time for tandem hops. 
However, there is as yet no adequate data for engineering otherwise. 


3.3 Determination of allowable hop length to meet an objective 


It is often desirable to be able to determine the maximum allowable 
hop length, for a given set of system parameters and location, for which 
the expected outage just equals the objective, without doing it by trial 
and error. This section describes a graphical procedure for determining 
the allowable hop lengths for the individual polarizations and approxi- 
mately for the weighted average. 

As mentioned in the previous section, it order to reduce the outage 
time the rain attenuation margin must be increased either by changing 
the equipment parameters or by changing the hop length or perhaps 
both. As can be seen from Fig. 3 by using the vertical polarization scale, 
increasing the fade margin by 3 dB by changing equipment parameters 
_but keeping the hop length at 40 km reduces the outage time from 11 
minutes to 8.5 minutes. This is not a significant reduction considering 
the difficulty involved in gaining 3 dB of margin by changing the wave- 
guide loss or antenna size. The more effective way of increasing the 
margin is to decrease the hop length because changes in both margin and 
hop length act together to decrease the outage time. Therefore, in cases 
where the hops are rain-limited, the best procedure is to use the best 
practical system parameters and adjust the outage by changing the hop 
length. 

If the system parameters are fixed, the rain attenuation margin varies 
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with hop length according to the equation 
Do 
Mr = Mro + 20 log (4) 


where Mp, is the rain attenuation margin on a hop length D,. The values 
of Mr, and D, therefore become a measure of the equipment perfor- 
mance of a hop, dependent on how that particular equipment has been 
engineered. In the previous examples, we have shown that Mr, = 50 dB 
on a 40-km hop is typical of the Western Electric 3A-RDS radio system. 
A plot of eq. (4) on the vertical or horizontal scales of a rain chart is called 
the system characteristic curve for the system with Mp, at D,. Figure 
5 shows such curves for Mp, = 50 dB at D, = 40 km. The system char- 
acteristic displays the rain outage time as a function of hop length by 
reading vertically upward from the system characteristic at the desired 
hop length to the rain outage curve. 

The outage objective for a given hop length, assuming the objective 
is prorated proportionally to the hop length, is 


D 
Tops/Hop = ToBs/ROUTE Geereeers (5) 
route length 


Using an objective of 0.02 percent per 400-km route, eq. (4) becomes 
Togss/Hop = .26 D minutes (6) 


when D is in km. If eq. (6) is plotted on the outage time scale of a rain 
chart as a function of hop length D along the system characteristic on 
either the vertical or horizontal scales, it becomes the system objective 
curve for the system. Figure 5 shows system objective curves for both 
polarizations and a system with Mr, = 50 dB at 40 km. 

Since the rain outage curve relative to the system characteristic curve 
is the hop outage as a function of hop length, and the system objective 
curve relative to the system characteristic curve is the hop objective, the 
maximum hop length, outage time, and corresponding fade margin can 
be found from the intersection of the two curves. Thus from Fig. 5, the 
values listed in Table IIA are found for the example system. 

The maximum allowable hop length for which the weighted average 
outage time (0.6 Ty + 0.4 Ty) meets the objective can be found ap- 
proximately by taking a weighted average of the vertical and horizontal 
allowable hop lengths: 


Dmax-ave © 0.4 Dmax-v + 0.6 DmMax-H (7) 


Note that the weighting is reversed because the higher outage of the 
horizontal polarization contributes more to the weighted average outage 
even with the 60-40 weighting. Once the maximum hop length has been 
calculated, the corresponding fade margin can be read from the system 
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Table Il — Allowable hop length calculations * 


A. Calculations for V and H polarizations 


Polarization 
V H 
Maximum hop length 24 km (15.0 mi) 18 km (11.3 mi) 
Rain outage time 6.2 min 4.7 min 
Rain attenuation margin required 54.5 dB 57.0 dB 
B. Calculations for weighted average outage 
Approximate maximum hop length [eq. (7)] 20.4 km (12.8 mi) 
Rain attenuation margin [eq. (45) 55.9 dB 
Vertical polarization outage 2.5 min 
Horizontal polarization outage 9.5 min 
Weighted average outage 5.3 min 
Objective for 20.4-km hop 5.3 min 


* For a system with a rain attenuation margin of 50 dB on a 40-km hop using the rain 
outage chart in Fig. 5. 


characteristic or calculated from (4), and the horizontal and vertical 
outage times can be read from the rain outage curve. Continuing the 
previous example gives the results listed in Table IIB, which are indi- 
cated in Fig. 5. 


3.4 Sensitivity of allowable hop length to rain attenuation margin 


Different types of radio systems have different attainable rain at- 
tenuation margins not only because the waveguide losses or antenna 
gains are different, but also because of inherent system performance 
capabilities. In this section we illustrate the sensitivity of the maximum 
allowable hop length to the rain attenuation margin by comparing the 
maximum allowable hop lengths of systems with different rain attenu- 
ation margins on a 40-km hop. 

Table III and Fig. 6 show the results of calculations paralleling those 
of Section 3.3 but for a system with Mp, = 45 dB on a 40-km hop. For 
both systems, the weighted average outage is equal to the outage ob- 
jective. 

The results show that a 5-dB decrease in rain attenuation margin re- 
quires a 16 percent decrease in hop length relative to the average of the 
two hop lengths. Calculations for other values of fade margins and other 
locations have shown that for fade margins ranging from 40 to 55 dB on 
a 40-km hop and for all cities where the maximum allowable hop lengths 
are 50 miles or less, a 5-dB difference in rain attenuation margin results 
in a difference of 16 to 18 percent in allowable hop length, and a 10-dB 
difference gives a difference of 30 to 35 percent in allowable hop length. 
Thus relatively small differences in margins can give substantial savings 
in system costs by reducing the number of repeaters required. 
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3.5 Effect of dynamic-range limited rain attenuation margin 


Reducing the hop lengths decreases the rain outage time for two rea- 
sons: because the rain attenuation margin increases, and because the 
amount of rain attenuation incurred decreases. In order to actually re- 
alize that part of the decrease due to the rain attenuation margin in- 
crease, the AGC range of the radio system receiver must be adequate. If 
the AGC range is inadequate, it will be unable to maintain a constant 
signal level in the receiver and an outage will be caused by loss of signal 
level rather than degraded signal-to-noise ratio. In such cases the system 
characteristic curve does not show a continual increase in margin as the 
hops are shortened, but remains constant at some limiting value. 

Figure 7 shows an example of a system with 45 dB of rain attenuation 
margin on a 40-km hop, but with a maximum rain attenuation margin 
of 48 dB, limited by the receiver AGC range. Figure 7 uses the vertical 
polarization only, but the same principles apply for both horizontal 
polarization outages and weighted-average outages. 

The limited AGC range decreases the maximum allowable hop length 
substantially. For example, in Fig. 7 if the AGC range were not limited, 
the maximum allowable hop length would be 15 km (10 miles) and a rain 
attenuation margin of 53 dB would be required. With the rain attenua- 
tion margin limited to 48 dB, the maximum allowable hop length is 12.5 
km (7.8 miles), or a decrease of 22 percent relative to 16 km. This would 
require a 28 percent increase in the number of repeaters. 

This effect is much more substantial in the Southeastern U.S. where 
the hops must be short with correspondingly large margins required. 


IV. EFFECTS OF THE VARIATIONS IN THE ANNUAL OUTAGE TIMES 


Lin‘ has discussed the variability of the rain rate distributions from 
year to year and has emphasized the need for stable statistics on which 
to engineer radio systems. In this section we discuss the implications of 
this variability on the reliability of systems engineered by the proposed 
methods and show the penalty for engineering for worst case statis- 
tics. 


4.1 Estimate of variability of annual outage times 


Figures 8 and 9 show rain charts with the worst (A) and best (E) annual 
distributions, and worst (B) and best (D) 5-year average distributions, 
in addition to the 20-year average distribution (C). Figures 8 and 9 also 
show the system characteristic and system objective curves for the ex- 
ample system with 50-dB rain attenuation margin on a 40-km hop. For 
simplicity, only the vertical polarization will be considered; similar 
conclusions would be drawn for engineering based on the horizontal 
polarization outage or the weighted-average outage. 
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Table Ill — Allowable hop length calculations* 


A. Calculations for V and H polarizations 


Polarization 


V H 
Maximum hop length 20 km (12.5 mi) 15.5 km (9.7 mi) 
Rain outage time 5.2 min 4.0 min 
Rain attenuation margin required 51.0 dB 


B. Calculations for weighted average outage 


Approximate maximum hop length [Eq. (7)] 17.3 km (10.8 mi) 
Rain attenuation margin [Kq. (4)] 52.3 dB 
Vertical polarization outage 2.3 min 
Horizontal polarization outage 7.8 min 
Weighted average outage 4.5 min 
Objective for 17.3-km hop 4.5 min 


* For a system with a rain attenuation margin of 45 dB on a 40-km hop using the rain 
outage chart in Fig. 6. 


First assume that the system has been engineered for the maximum 
allowable hop length for which the outage on the vertical polarization 
will meet a 0.02 percent per 400 km objective. The resulting hop length 
in Fig. 8 is 19.5 km (12.2 mi) and the average annual outage is 5 minutes 
per year based on the 20-year average distribution. 

The curves shown on the rain outage chart are actually those distri- 
butions which were measured over the 20-year base period, 1953 to 1972. 
Thus the outage times read from the rain chart are those outage times 
which would have been measured if a system had been operating during 
the 20-year base period (assuming the rain theory is correct); but they 
are probably not the outage times that will be measured in the next or 
any other 20-year period. They are, however, the best estimate of what 
similarly averaged outages would be for any 20-year period. Furthermore, 
the outage time indicated by the 20-year average curve is the best est1- 
mate of what the annual outage time will be in any one year although 
we know that it probably will not be that value. 

The annual outage times indicated by curves A and E in these figures 
give some indication of the extreme values that can be expected over a 
20-year period. In Fig. 8, the largest outage time is about 17 minutes, a 
little over 3 times the design value; the smallest outage time is much less 
than one minute. The largest annual outage time averaged over any 5- 
year period is expected to be about 8 minutes; and again the smallest is 
much less than 1 minute. Similar results are obtained from Fig. 9. 

A similar ananysis was done for each of 13 representative cities in- 
cluding those in Figs. 3 to 9 and the results are listed in Table IV. Table 
IV is divided into two parts. For those cities listed in part B the allowable 
hop lengths are so long, and the corresponding rain rates so low, that 
meaningful short-term distributions at high rain rates could not be 
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Table IV Factors by which the 20-year average outage time on the 
vertical polarization is exceeded* 


Factor by which outage time 
exceeds 20-yr average 


20-yr averge outage time 


Rain Outage Hop length, outage time, l-yr 5-yr 5-yr l-yr 


Chart km miles minutes max max min min 
(A) 
Fig. 4 32.3 20.2 8.4 2.6 1.3 0.6 f 
Figs. 5, 6 23.9 14.9 6.2 7.1 1.9 0.3 0.0 
Fig. 7 19.2 12.0 5.0 4.8 2.4 0.1 0.0 
Fig. 8 19.5 12.2 5.1 3.2 1.6 0.0 0.0 
Fig. 9 17.7 11.1 4.6 6.3 1.7 0.5 0.0 
(Not shown) 30.8 19.2 8.0 5.0 1.8 0.5 t 
(Not shown) 23.1 14.4 6.0 2.5 1.7 0.6 0.0 
(B) 
Fig. 3 36.2 22.6 8.2 3.9 2.3 0.3 + 
(Not shown) 36.2 22.6 5.9 5.8 1.8 0.5 t 
(Not shown) 36.2 22.6 3.7 4.1 1.8 + t 
(Not shown) 36.2 22.6 2.0 10.0 2.2 + + 
(Not shown) 36.2 22.6 2.8 71 2.1 + f 
(Not shown) 36.2 22.6 1. 7.1 3. t + 


* By the maximum and minimum 1-year outage times and 5-year average outage times. 
Part A uses representative cities for which the hop length listed is the maximum allowable 
to meet the outage objective. Part B uses representative cities for which the hop length 
is shorter than the maximum because data was not available at the maximum allowable 
hop length. The outage time allowable at the 36.2-km hop length is 9.4 minutes. A system 
with 50-dB margin on a 40-km hop is assumed. 

t Data not available. 


generated. Consequently, the calculations were made at the longest hop 
length for which data was available—36.2 km. 

The data in Table IV show that for the midcontinent cities in part A 
the factors by which the maximum 1-year outages exceed the engineered 
value range from 2.5 to 7.1. Factors by which the maximum 5-year av- 
erage annual outages exceed the engineered value range from 1.3 to 2.4. 
At every location there should be at least 1 year out of 20 for which there 
is no outage. The variability in the outage for the cities in part B is 
slightly more, the maximum 1-year factors ranging from 3.9 to 10, and 
the maximum 5-year factors ranging from 1.8 to 3.4. 

The question of whether a hop or a route is performing as engineered 
inevitably arises. Two additional factors which affect the observed 
outages of a route must be considered. First, as demonstrated by Lin,* 
the outage of a route consisting of several hops should not be as variable 
as the individual hops themselves. Lin’s Fig. 16 shows roughly a factor- 
of-2 reduction in his At/t factor, which is equivalent to the factor listed 
for the 1-year maximum in Table IV,* for a route consisting of three 

* In Lin’s paper At is the worst-year minutes less the best-year minutes. However, the 


best-year minutes are negligible, so the ratio is essentially worst year minutes divided by 
the 20-year average which is the same as used in Table IV. 
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hops. Secondly, the route outage should not be as large as the sum of the 
individual hop outage because of joint fading on tandem hops. This effect 
should be more important in the midcontinent cities where the hops are 
shorter. 

Based on the foregoing, the following guidelines seem reasonable. First 
it must be definitely established that the outage in question is caused 
by aerial attenuation by rain. Then, if the route outage time of a route 
containing three or more hops exceeds the engineered value by more than 
a factor of 5 in any one year, or by a factor of 2 for any 5-year average, 
the rain outage is excessive and the reason for the excessive outage should 
be determined. If the outage time of a single hop or two tandem hops 
exceeds the engineered value by more than a factor of 10 in any one year, 
or a factor of 4 for any 5-year average, the reason for the excessive outage 
should be determined. 


4.2 Engineering for worst-case outages 


To avoid exceeding the outage objective for any one year, or for any 
5-year average, would require that the hops be engineered so that the 
objective is met for the estimated maximum 1- and 5-year average annual 
outages respectively. Figures 8 and 9 illustrate the procedure for the 
example system using only the vertical polarization outage. 

In Fig. 8 the hop length is 19.5 km (12.2 miles) based on the 20-year 
average, 17.7 km (11.1 miles) based on the maximum 5-year average 
annual outage time, and 12.9 km, (8.1 miles) based on the maximum 
1-year annual outage time. These hop lengths are 9.2 and 33.9 percent 
reductions in hop length, which in turn mean 10.1 and 51.3 percent in- 
creases in the number of repeaters, respectively. 

Table V lists similar percentages for eight Eastern and Midwestern 
cities. Such comparisons are not meaningful for the far Western cities 
because the allowable hop lengths based on 20-year average outage times 
are much longer than are used in practice. (In other words, the hops are 
not rain-attenuation limited.) Table V shows that the percentage in- 
crease in number of repeaters ranges from 9 to 27 percent if the hop 
lengths are based on maximum 5-year average outage times, and from 
25 to 77 percent if the hop lengths are based on the maximum 1-year 
outage times. Because radio repeaters are so expensive, such increases 
in the number of repeaters could make rain-attenuation-limited radio 
systems very uneconomical. 


V. GEOGRAPHICAL COVERAGE 


Although charts have been produced for many cities, there are still 
areas a few hundred miles on a side for which no rain data exists, and so 
the problem of how to engineer radio systems in these areas still exists. 
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Table V — Percentage decrease in allowable hop lengths and resulting percentage increase 
in number of repeaters* 


Percentage increase in number of repeaters 


ZL6L YAEWSAON “IWNYNOP WOINHOSL WSLSAS 114d SHL 9291 


Hop length for Percentage decrease in hop length for hops for hops engineered to indicated 
Rain outage EUVESUAVERAR®.: . us “EHBING Ore? romndicated Cistbuslonye. cisions 
chart km miles 5-yr max l-yr max 5-yr max 1-yr max 
Fig. 3 36.5 22.8 21.1 39.0 26.7 63.8 
Fig. 4 32.3 20.2 8.3 40.5 9.1 68.0 
Figs. 5,6 23.9 14.9 11.3 43.6 12.7 77.1 
Fig. 7 19.2 12.0 — 20.0 — 25.0 
Fig. 8 19.5 12.2 9.2 33.9 10.1 51.3 
Fig. 9 17.7 11.1 19.8 30.5 24.7 43.9 
(Not shown) 30.8 19.2 16.3 32.5 19.4 48.2 
(Not shown) 23.1 14.4 18.1 33.4 21.7 50.2 


* Resulting from engineering the maximum 1- and 5-year average outage times to meet a 0.02 percent per 400-km objective. 


At the present time there is no definitive proven solution to this problem, 
but the following approaches seem reasonable. 

The problem can be approached in two basic ways: interpolation be- 
tween locations where data exists, or identification of the unknown 
location with.a known location based on consideration of climates and 
local judgment. Usually some combination of these two approaches is 
the most satisfying intuitively. 

There is no reason to suppose that other than linear interpolation. 
should be used. Linear interpolation can be used by calculating the 
outage times for a given system at different locations and interpolating 
between them, or by calculating the allowable hop lengths and inter- 
polating them. The main advantage interpolation has over JMGE MERE is 
that it is consistent and reproducible. 

In using judgment of climatological conditions it is of the utmost im- 
portance to remember that it is the rainfall rate that determines outage 
time and not the total amount of water that falls. The northwest coast 
of the United States is the primary example of a very wet region where 
there is virtually no rain-attenuation-caused outage. Large scale 
climatological factors which seem to bear some relation to high rain rates 
are number of thunderstorms, late summer humidity, and total July 
precipitation. These are probably related because most of the rain rates 
which are large enough to cause an outage are due to thunderstorms. For 
example, total July precipitation is related to thunderstorms because 
in July most of the precipitation is from thunderstorms. Terrain should 
also be considered, especially in the lee of mountains, because rough 
terrain and mountains contribute to the formation of thunderstorms. 

Finally, local knowledge and judgment should be used in comparing 
the area in question to a location where data is available. 
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Optimum Digital Filters for Interpolative A/D 
Converters 


By A. N. NETRAVALI 
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Interpolatine analog-to-digital (A/D) converters allow a fine rep- 
resentation of signals by making many coarse representations and 
averaging them using a digital filter. In this paper, we give a method 
of optimizing the characteristics of this digital filter under two different 
criteria. The first criterion is the well-known signal-to-noise (S/N) 
ratio, whereas the second criterion is the weighted sum of the signal 
power, the quantization noise power, and the noise power within a given 
band of frequencies. We design optimum digital filters and simulate 
their performance on the computer. We show that the theoretically 
predicted S/N ratio is in good agreement with the performance ob- 
tained by computer simulation. It is seen that about 23 dB improvement 
in S/N ratio over the S/N ratio attainable by a constant-weight digital 
filter is possible when the number of coarse quantizations is 256. We 
also study the effects of changing various parameters of the A/D con- 
verter on the S/N ratio. 


|. INTRODUCTION 


Interpolative A/D converters! achieve a fine quantization of signals 
by making several coarse quantizations and averaging them. This re- 
quires high-speed operation of that part of the A/D converter which 
obtains the coarse quantizations. Higher and higher speeds are required 
for finer and finer ultimate quantization. This trade-off between the 
speed of operation and amplitude resolution is particularly relevant and 
important with present-day integrated circuit technology, which pro- 
vides high-speed operation but no high-amplitude precision. 

Several well known methods of obtaining the many coarse quantiza- 
tions exist. Goodman,! and Goodman and Greenstein,” have considered 
the ordinary delta modulator which gives a two-level representation of 
the signal at a rate many times higher than the Nyquist rate. The output 
of the delta modulator is filtered by a digital filter and resampled at 
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Fig. 1—An interpolative A/D converter. 


Nyquist rate to obtain the PCM output. The performance of such an A/D 
converter depends upon the speed of the delta modulator and the 
characteristics of the digital filter. 

Another method of obtaining the coarse quantization has been pro- 
posed recently by Candy.? In this method, the coarse quantizations are 
obtained by a direct feedback encoder shown in Fig. 1. In this encoder 
a difference between the analog input and a coarsely quantized repre- 
sentation is filtered by an analog filter with characteristics H(s), and then 
quantized. This is done at speeds higher than the Nyquist rate. The 
output of the quantizer is represented by binary words and filtered by 
a digital filter having characteristics D(z). The output of the digital filter 
is resampled at a slower rate to obtain the final digital output at the 
Nyquist rate. Use of direct feedback encoding allows shaping of the 
quantization noise in such a way that the digital filter can be made very 
simple. Candy? has shown that when the analog filter is taken to be a 
pure integrator the simple digital filter corresponding to “accumulate- 
and-dump” performs adequately. 

Candy et al.* have described a method of optimizing the weights of 
the digital filter when the analog filter in the ‘fast loop” is a pure inte- 
grator. They have shown that the optimum weights can be approximated 
by aset of triangularly distributed weights and evaluated the improve- 
ments in S/N ratio by using these weights. Their results are applicable 
only when the integrator in the “fast loop” is not reset to zero at the 
beginning of each slow cycle. In this paper, we first show that when the 
analog filter in the “fast loop” is reset to zero at the beginning of every 
slow cycle an advantage in the S/N ratio is obtained when the uniformly 
distributed weights are used. We then give a different method of opti- 
mizing the digital filter characteristics under the assumption that the 
analog filter is reset. Our method of optimization is applicable to the case 
of any arbitrary analog filter in the place of the integrator in the fast loop. 
The resulting optimum weights when the integrator is reset every slow 
cycle have a different shape than the optimum weights given by Candy4 
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which are applicable when the integrator is not reset. We compare our 
optimum weights with the triangular weights proposed by Candy as an 
approximation to his optimum weights. Optimization of the digital filters 
using a different criterion, which includes a deviation of the digital filter 
characteristics from desired characteristics is also discussed. In this case 
it is possible, for example, to shape the discrete Fourier transform of the 
digital filter weights so that it resembles, as far as possible, an ideal 
low-pass filter. We evaluate the performance of the A/D converter in 
terms of S/N ratio by computer simulation for several typical cases. 


ll. SUMMARY OF RESULTS 


Our computer simulations indicate that there is about 3 dB im- 
provement in S/N ratio by resetting the integrator at the beginning of 
each slow cycle when uniform weights are used for the digital filters. This 
improvement is independent of the coarseness of the quantizer in the 
fast loop, the number of fast cycles and the correlations present in the 
input signal. The use of optimum weights for the digital filter leads to 
significant improvements in S/N ratio over that obtained by a digital 
filter with constant weights. This improvement although independent 
of the coarseness of the quantizers depends on the number of fast cycles; 
for 32 fast cycles, there is about a 14 dB improvement, whereas for 256 
fast cycles, there is a 23 dB improvement. Also, the optimum weights 
outperform the “triangular” weights used by Candy et al.* by about 7.30 
dB when the number of fast cycles is 32 and by about 8.80 dB when the 
number of fast cycles is 256. We also show that there is a good agreement 
between the theoretically predicted S/N ratio and that obtained from 
computer simulations of the A/D converter. Changing the analog filter 
from an integrator to a general analog filter with a given characteristic 
indicates that there is a gain of a few dB in S/N ratio by choosing the de 
gain and the cutoff frequencies judiciously. Our second method of op- 
timizing the digital filter characteristic allows us to minimize the de- 
viation of its frequency characteristics from a given characteristic. Using 
the desired characteristic to be ideal low-pass, we are able to decrease 
the noise power in a given band of frequencies. This decrease is about 
0.5 to 1.0 dB, but it comes at the expense of an increase in the overall 
noise power of about 1.0 to 1.5 dB. Thus the digital filter suppresses the 
noise power in one band of frequencies, but enhances the noise in the 
rest of the frequency band, resulting consequently in an overall increase 
in the noise power. 


Ill. DERIVATION OF OPTIMUM DIGITAL FILTER WEIGHTS 


In this section, we derive the weights of the optimum digital filter. First 
we concern ourselves with those digital filters which minimize the S/N 
ratio, and then derive those weights which can be spectrally shaped. 
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Let x(t) be the analog input to the A/D converter shown in Fig. 1. Also 
let h(-) be the impulse response of the time-invariant analog filter in the 
fast loop; N, the number of fast cycles; T, the fast sampling period; and 
q;, the output of the quantizer at the jth fast cycle. We assume that the 
output of the digital-to-analog converter is given by 


q(tt)=qj JT St<GUt IT (1) 
The equation for the fast loop can be written as: 
t 
f, h(r)[x(t — 7) — q(t — 7) ]dr = q(t + T)+n(t + T) (2) 
0 
Here we have assumed that the analog filter is reset at the beginning of 
each slow cycle and that the quantization distortion can be represented 


by additive random noise n(-). Assuming that x(t) is constant (=x) over 
aslow cycle, then at t = (1+ 1)T, 


iT iT 
P f, h(t)dt — f, hig? — r)dr = q(G+ DT] 4+n[G4+ DT] 


(3) 
Now letting 
(k +17 
f, h(t)dt = hp (4) 
kT 
eq. (8) can be written as: 
i-1 i-1 
xD hp - D GQi-r-the = 3 + 07; b= diay lV (5) 
k=0 k=0 
where 
nj = ntiT) 
Equation (5) can be written fori = 1,...,.N, in a matrix form 
ho 
hy 
x-A=HQ+No+dqo} (6) 
hn-1 
where 


N-1 
eel (ho, ho + hy, > hi) 
1=0 
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1 0 0 

he 1 0 0 

hy ho 1 0 
H=|. ; | 

hy-2 hn-3°° hyho | 


No = col (ny, no,..., nN) 
Q = col (q1, G2,.--, GN) 


Observe that matrix H has an inverse and therefore eq. (6) can be re- 
written as: 


h 
gest Nyaa] : | (7) 
hn-1 


The digital filter will process vector Q every slow cycle by multiplying 
it by a weight vector D, and thus the PCM output will be 


ho 
oar aiarin atti) , | (8) 
An-1 


Here we assume that DTU = 1, where U = col (1, 1,..., 1). The first term 
on the right-hand side of eq. (8) is the signal component, whereas the 
second term is the noise component. The third term results from the 
initial condition on the D/A, go. We assume go = 0. In order to maximize 
the ratio of signal energy to the noise energy, we maximize the following 
expression: 


(S/N)*(D?G~1A)?/(DTH~1No)? (9) 


where (-) denotes expectation. Assuming? that the noise components n; 
are independent, identically distributed with variance o?, we can write 
eq. (9) as: 


(S/N) = - [DTH-!AAT(H-})TD]/DTH-\(H-TD (10) 
oO 


We note that since H has an inverse H~!, (H—1)7 is positive definite and 
therefore the denominator of the right hand side of eq. (10) will not be 
zero unless D = 0, a case which we rule out. This implies that (S/N) will 


t This assumption is not required. It is easy to extend the following analysis to colored 
noise. 
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be bounded from above. Since (S/N) is a ratio of two quadratic forms 
generated by two symmetric matrices, we can write the optimum D, D*, 
as a solution of the eigenvalue problem 


(H~1A)(H~1A)TD* = Xnax(H~1)(H~1)7D* (11) 
or 
HTAA?T(H~-1)TD* = XmaxD* (12) 


where \max is the maximum eigenvalue. It is easy to see that the only 


eigenvector for eq. (12) corresponding to a nonzero eigenvalue is given 
by 





D*=HTA (13) 
for which 
T 
(s/n) = 4 4 
oO 
1[N /i-i 2 
G2 E (= hr) 


Writing out H and A, we get 


N-2 jJt+1 
Lh ( 5 he) 
jJ=-1 k=0 


N-3  _j+2 
D* = Yh; ( sz hn) 
k=0 


gerd 


N-(N+1) j+N 

yy (> he) (15) 
j=l k=0 

where we have assumed for notational convenience that h_, = 1. If the 

filter in the fast loop is a pure integrator, then h; = T, and the optimum 

digital filter can be written as: 


D = col (Di, saris Dj, ghee Dn) 
where 


T(N -]j j ; 
py=PNRJtDNHD jay com 
and for large N the S/N ratio is given (except for a proportionality 
constant) by 


_ NN + 1)QN +1) 


S/N 
; 6 


(16b) 
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3.1 Optimum digital filter with spectral shaping 


Let D(w) be the discrete Fourier transform of the samples 
{Dzle=0,...,N-1 and C(w) be the transform of the desired response that 
mene Nn-1- The shaping of the 
digital filter in the Fourier domain can thus be accomplished by proper 
choice of C(w). We use the following expression for the error between 
the two: 


Err = S|[D@) — C()]|? dw 
= f[D(w) — C(w)|[D(w) — C(w)]* dw 
= SD(w)D*(w) dw — SD(w)C*(w) dw 
— f(D*(@)C(w) dw + f[C(w)C*(w) dw (17) 


where (-)* is the complex conjugate. In minimizing Er with respect to 
D, we can drop the third term of eq. (17) and rewrite (17) as 


N-1 N-1 ; 
Err= > D? = f( > Dye~rohiN) C*(w) dw 
k=0 k=0 


_ f ee Cpe] D*(w) dw 


h=0 
N-1 N=-1 
= Di-2 X% D.C, 
h=0 h= 
= DTD — 2D™C (18) 


The performance function (PF) that we want to maximize can be 
written as: 
(PF) = D?TH-!A(H-!A)7D — XDTH-\(H-4)7D — y(DTD — 2D'C) 
(19) 
where the first term on the right-hand side corresponds to signal energy, 
second term corresponds to noise energy and the last term is the Erg 
from eq. (18), and A and ¥ are positive constants. Equation (19) can be 
rewritten as: 
(PF) = D’[H-!A(AH—!A)? — \H-Y(A-? — yIJD + 2yDITC 
(20) 
The best D which maximizes (PF) is given by 


D* = 2y [H~1A(H—1!A)? — AN“)? — yT]HIC (21) 


IV. RESULTS OF COMPUTER SIMULATION 


In our computer simulations we used uniformly distributed pseudo- 
random noise as the input signal x(t) to the A/D converter. This was held 
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re 2—Performance of an interpolative A/D converter with various digital filter 
weights. 

constant throughout each slow cycle. We also considered cases when the 
input signal was filtered by an appropriate filter before going into the 
A/D converter. Simulations were carried out with the quantizer having 
two different step sizes, namely 0.125 and 0.0625 (signal range 0-1). The 
quantizer was assumed to have an unlimited number of levels and thus 
the effects of saturation were neglected. This assumption becomes more 
restrictive when the gain of the analog filter in the fast loop is increased. 
To evaluate the dependence of S/N on the number of fast cycles, several 
(4, 8, 16,...,256) values of fast cycles were used. For the purpose of 
comparison, we also considered the following cases: 


(1) Uniform weights, i.e., Dj; =1,j =1,...,N, with integrator not 
reset. 

(i) Triangular weights, i.e.,D; = min J, N+1-—j),j =1,...,N, with 
integrator not reset. Both these weights have been investigated pre- 
viously.?4 
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Fig. 3—Improvements in S/N ratio over constant weight digital filter. 


4.1 Effect of integrator reset with uniform weights 


The effect of resetting the integrator in the fast loop was evaluated 
by using uniform weights for two cases: (i) integrator reset, (ii) integrator 
not reset. The resulting S/N ratios are plotted in Fig. 2. The improve- 
ment in S/N ratio by resetting the integrator is plotted in Figs. 3 and 4, 
for two quantizer step sizes. It is seen that there is about 3 dB improve- 
ment by resetting the integrator, and this improvement is somewhat 
independent of the quantizer step size and the number of fast cycles. This 
can be easily explained by rewriting eq. (8), for h; = 1 and D; = 1, as: 


N 
2X 4 = Nx — nn ~ go (22) 


Thus there is an extra term on the right-hand side, qo, if there is no reset. 
Assuming that it is comparable to ny, and that it is not correlated with 
nn, the S/N ratio would decrease by about 3 dB due to its presence. We 
also simulated the effects of correlations in the input data, by filtering 
the pseudorandom noise, and then putting it through the A/D converter. 
Several low-pass filters were tried, and it was observed that the im- 
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Fig. 4—Improvements in S/N ratio over constant weight digital filter. 


provement in S/N ratio was still 3 dB regardless of the amount of low- 
pass filtering. 


4.2 Effect of optimum digital filter weights 


Figure 2 shows the effects of optimum digital filter weights on the S/N 
ratio. The analog filter in the loop is assumed to be a pure integrator with 
unity gain and it is reset at the beginning of each slow cycle. Figure 2 also 
shows the advantages of using the triangular weights, proposed by Candy 
et al., when the integrator is not reset. As observed by Candy et al., tri- 
angular weights are significantly better than the uniform weights, and 
the optimum weights allow a further increase in S/N ratio over the tri- 
angular weights. Figures 3 and 4 show the improvements in S/N ratio 
over those obtainable by the uniform weights when the integrator is not 
reset. It is seen that the rate of change of S/N ratio depends upon the 
number of fast cycles and is in close agreement with that predicted by 
eq. (16b). The S/N ratio using uniform weights when the integrator is 
not reset is given by (except for a proportionality constant) — 


S/N = 7 (23a) 
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Fig. 5—Weights of optimum- and triangular-weight digital filters. 


and with triangular weights 


_(N+t 1)8 


S/N 6 


(23b) 


These are derived by Candy et al.4 Our simulations are in close agreement 
with the above equations. Thus when N = 32, the improvement in S/N 
ratio by using triangular weights over uniform weights is 6.55 dB, which 
is close to 6.40 dB predicted by the above equations. Similar agreement 
is found at other values of N. Also for large values of N the improvement 
obtained by our optimum weights, in the presence of integrator being 
reset, over the triangular weights without resetting the integrator is about 
7.25 dB as predicted by eqs. (23b) and (16b). Our simulations indicate 
that this improvement varies between 5.80 and 10.0 dB with a mean of 
7.57 dB. This is a little higher than that predicted by the equations, 
however the agreement is satisfactory. 

The weights of the optimum digital filters are shown in a graphical 
form in Fig. 5 along with the triangular weights used by Candy et al.4 It 
is seen that they have a parabolic shape. Although we have not consid- 
ered the effect of approximations, for implementational simplicity, the 
filter shape could be approximated by piecewise straight lines. 
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4.3. Effect of variation of the analog filter in the fast loop 


We attempted to evaluate the effect of varying the analog filter in the 
fast loop on the S/N ratio. It is known that certain types of analog filters 
tend to make the fast loop unstable; however, we did not consider 
questions of stability. Two types of transfer functions for the analog filter 
were considered: 


— _ oe _ 
His) s+B 
and 
oats SU a 
Has) = s(s + B) 


The first case resulted in h; = ae—'8 (using eq. 4) and the second case 
gave h; = a — ce—'8, where constants a, 6, and o are related to @, B. 
Several simulations were run by varying a, 8, and o. For each of these 
simulations, the optimum weights were computed by eq. (15), and the 
resulting S/N ratio was compared with that obtained by using a pure 
integrator in the fast loop and the optimum digital filter. We considered 
only 32 fast cycles and a quantizer step size of .0625. In the first case, it 
was found that larger a and smaller 6 generally gave better S/N ratio. 
At a = 1.2 and 8 = 0.01, the improvement in S/N ratio was about 3.0 dB. 
For many other cases studied, the improvement was somewhat marginal. 
For the second case, again, larger values of a, smaller values of 6 and o 
around 1.0 gave the best results. At a = 1.8, 6 = 0.01, and o = 1.0, the 
improvement in S/N ratio was about 4.2 dB over that obtained by pure 
integrator in the fast loop. Thus it appears that S/N ratio can be further 
improved by a proper choice of the analog filter in the loop. 


4.4 Effect of spectrally shaped digital filters 


Our final simulations used digital filters which resemble a given digital 
filter as far as possible. For our simulation we obtained the desired digital 
filter characteristics from an analog function C(t) whose Fourier 
transform C(f) was 0 outside |f| > Q and was constant (=Mag) in the 
interval |f| < 2. Sampling such a function at N times the Nyquist rate 
(corresponding to the number of fast cycles) gave 


_ 2+Mag-Q-N 


TL 


sin (ix/N) i=0,...,.N-—1 

Using these weights for the desired filter characteristics and some values 
of d, y (of Section 3.1), optimum digital filters with spectral shaping were 
obtained for the case when the analog filter in the fast loop was a pure 
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integrator. Computer simulations were carried out for various values of 
Nand y, N = 82, and quantizer step size = .0625. Two quantities were 
measured: (i) S/N ratio as before, (ii) the noise power in frequency band 
—Q to +Q. It was found that by giving a high value to y (i.e., heavily 
penalizing any deviation of the filter characteristics from the desired 
characteristics), a decrease of about 1 dB in the noise power in frequency 
band —Q to +Q was possible. However, this resulted in a decrease of S/N 
ratio by about 1.5 dB. Thus it appears that the inband noise could be 
suppressed to some extent at the expense of decrease of overall S/N 
ratio. 


SUMMARY AND CONCLUSIONS 


In this paper, we have given two techniques for optimizing the digital 
filter characteristics of an interpolative A/D converter. Computer sim- 
ulations showed that the optimum digital filters with the integrator reset 
increases the signal-to-noise ratio by as much as 23 dB over that ob- 
tainable by a digital filter with uniform weights and no resetting of the 
integrator. We also showed that by resetting the integrator a 3 dB ad- 
vantage in signal-to-noise ratio is obtained when uniform weights are 
used. We varied the transfer function of the analog filter in the fast loop 
and found that a gain of a few decibels is possible by proper choice of the 
analog filter. Finally we considered digital filters whose characteristics 
could be made close to certain desirable characteristics, and found that 
it is possible to decrease the quantization noise power within a band, but 
only at the expense of decrease of the overall signal-to-noise ratio. We 
note that two important factors, which we have not paid attention to, 
are: (i) stability of the fast loop, and (ii) simplicity of implementation 
of the digital filters. These would be crucial in any practical implemen- 
tation of the interpolative A/D converters. 
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Construction for Group-Balanced Connecting 
Networks 


By F. K. HWANG and T. C. LIANG 
(Manuscript received April 1, 1977) 


We generalize the concept of balanced network to group-balanced 
network. An s-stage network ts called a group-balanced network if its 
input switches can be partitioned into groups and its output switches 
into groups such that the connection pattern (called channel graph) 
between an input group and an output group is independent of which 
groups we choose. We show by construction that under a simple div- 
istbility condition, a group-balanced network can be constructed sat- 
isfying the following requirements: (i) the number of stages is specified, 
(ii) the size of the switches in each stage is specified, (iii) the channel 
graph between an input group and an output group is specified. 


1. INTRODUCTION 
An s-stage (connecting) network satisfies the following conditions: 


(1) The network is composed of switches and links. Switches are ar- 
ranged in a sequence of s stages. 

(it) The switches in a given stage are identical. In particular, they have 
the same size, i.e., the same number of input terminals and output ter- 
minals. 

(iit) Links can exist only between two switches in adjacent stages. 


In this paper, we assume that each switch is a rectangular (matrix) 
switch; i.e., there is a crosspoint connecting every input terminal with 
every output terminal of that switch. Figure 1 illustrates a three-stage 
network. 

Consider an s-stage network and let S; denote a switch in the ith stage. 
Consider the paths in the network which connect an 8, (input switch), 
say, the kth, with an S, (output switch), say the jth. Taking the union 
of all such paths and replacing each switch on a path by a node, we have 
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Fig. 1—Three-stage network. 


the channel graph G(k, j) for that pair of switches. Suppose the collec- 
tion 


{G(k, J):J7 = 1,2,...} 
is identical for all k, and the collection 


{(G(k, j):k = 1,2,...4 


is identical for all j, i.e., the network is symmetric with respect to the 
switches in the first (last) stage. Then the network is called a partially 
balanced network. If, furthermore, G(k, j) is identical for arbitrary k 
and j, the network is called a balanced network.46 

In this note we generalize the concept of balanced network. An s-stage 
network is called a group-balanced network if its input switches can be 
partitioned into groups and its output stages into groups such that the 
connection pattern between an input group and an output group is in- 
dependent of which groups we choose. This connection pattern will again 
be referred to as the channel graph between the two groups. When both 
the input group and the output group contain a single switch, a group- 
balanced network reduces to a balanced network. Moreover, an s-stage 
group-balanced network can always be augmented into an (s + 2)-stage 
balanced network by adding a stage before the input stage in such a way 
that the switches in one input group always connect to the same set of 
switches in the new stage, and by adding a stage after the output stage 
with similar connections. We also note that every s-stage network can 
be viewed as a group-balanced network if all input switches are consid- 
ered to form one input group and all output switches to form one output 
group. 

The problem of constructing a balanced network with a specified 
channel graph and given switch sizes has been studied in Refs. 1-8. In 
this note we give a construction for group-balanced networks under 
similar conditions. 
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ll. A CONSTRUCTION 


For a given node in a channel graph G, we will call the number of links 
connecting it to a preceding stage its indegree, and the number of links 
connecting it to a succeeding stage its outdegree. We assume the speci- 
fied channel graph is regular in the sense that every node in the same 
stage has the same indegree and outdegree. Let d; and c; denote the in- 
degree and outdegree for a node in the ith stage, i = 1,...,s. We want 
to construct a group-balanced network B whose channel graph is the 
specified one and whose ith stage switches are of given size nj X m; (nj; 
input terminals and m; output terminals), i = 1,...,s. Note that the 
number of switches in the ith stage, say 1;, is completely determined 
from 


i-1 st1 : 
i= [[m; [J nj/d,i=1,...,8 
j=0 ~ j=iti 
where mo(ns+1) is defined to be the number of input (output) switches 
in an input (output) group and A is the number of paths from the first 
stage to the last stage in the specified channel graph. 


Theorem 1: Suppose d; divides n; and c; divides m; for everyi = 1,..., 
s. Then the desired B exists. 


Proof: The proof is by construction. Without loss of generality, we may 
assume that the number of stages s is even. For if s is odd, we can always 
add an (s + 1)th stage, which has a single node, to the channel graph by 
connecting that node with every node in the sth stage. Since s + 1 is now 
even, we can construct an (s + 1)-stage group-balanced network (by 
defining n,+, and m,4, properly) and then delete the (s + 1)th stage. Our 
construction is by induction on s, (s = 2, 4,6,...). 

Let S; denote a switch of size n; X m;. Fors = 2, take no/d> groups of 
switches S; and m,/c; groups of switches So. Connect every group of $1 
to every group of S» according to the specified channel graph G. The 
resulting network is the desired one. 

Next, consider an s-stage channel graph G for even s. Let f; be the 
number of nodes in the ith stage of G. Furthermore, let G' be the 
subgraph obtained from G by deleting its first and last (i — 1) stages. 
Suppose by induction, we have constructed an (s — 2)-stage network B’ 
with the specified channel graph G2. We show show to construct the 
s-stage network with the specified channel graph G. 

Take (n,/d,;) - (m/c,) copies of B’ and label them by B’ (i, 7) where 
i=1,...,n;/d, andj =1,...,mj,/c1. Note that the input (output) 
switches of B’ can be decomposed into g2(g,-1) groups each of which 
consists of fo(f,;-1) switches and the channel graph between every input 
group and every output group is G2. Take g; = (n,/d;) + (n2/da) - g2 
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Fig. 2—Specified five-stage channel graph G. 


groups (each containing f; switches) of S; and label the groups by F(u, 
v, w) whereu =1,...,n;/d;,v =1,...,n9,dgandw =1,...,g9. Connect 
F(u, v, w) to the wth input group of every B’(u, J) according to the con- 
nection of nodes in the first two stages of G’. Similarly take g, = (m/c ) 
- (Ms—1/Cs—1) * 8s—1 groups (each containing f, switches) of S, and label 
the groups by H(x, y, z) wherex =1,...,m/c1,y =1,..., (Ms—1/¢s—1) 
and z =1,...,s—1. Connect H(x, y, z) to the zth input group of every 
B’(i, x) according to the connection of nodes in the last two stages of G’. 
It is easy to verify that the channel graph between every F(u, v, w) and 
every H(x, y, z) is the graph G. 





Fig. 3—One of many possible ways of connecting groups, denoted by B®. 
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lll. EXAMPLES 


In this section we illustrate with several examples the scope of appli- 
cability of the construction method given in the last section. 


Example 1: Let the specified five-stage channel graph G be the one in 
Fig. 2 (solid lines only), where the specified switch sizes are 


n;= 1, No = 3, ng = 2, ng = 4, n5=4 
m, = 2, Mo = 5, Ms = 2, m4 = 4, ms = 1 


Construction: Since the number of stages is odd, we add an artificial 
stage (broken lines). We first construct G? which has f3 = 5 input nodes 
and f4 = 5 output nodes. Since 

m1 and “4=2 
C3 d4 


we take g3 = 2 groups of S3, g4 = 1 group of S4 and connect each group 
of S3 with the group of S4 according to G3. There are many possible ways 
of connecting, one of which is shown in Fig. 3 and denoted by B?. 


Next we construct G? which has two input nodes and five output 
nodes. Since 








m 
meey "ag Mog M24 
cao ds C4 d3 
we take 
ue ie 2 copies of B? 
ds C2 
Nn5 ng 
= —.—.g3 = 4 groups ofS 
82 ana. §3 group 2 
and g5 = Le groups of S5 
Co 64 


and make connection between groups according to G2. One possible 
connection, denoted by B?, is given in Fig. 4 (solid lines). 

Finally, we construct G! whose output stage can be ignored (since it 
is artificial) as long as we define ms = ng = 1. Take 


ne 


7 1 copy of B? 
dg C1 





and g; = £9 = 12 groups of S; 


7 ae 
The final product is given in Fig. 4 with broken lines added. 
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Fig. 4—A possible connection between groups according to G2. 


Example 2. Consider the channel graph in Fig. 5. Suppose we specify 
ny =1,n9=1,Nn3 = 2, my = 3, Me = 2, m3 = 1. Then our construction fails 
since m, is not divisible by c;. However, a balanced network having these 
parameters and the specified channel graph does exist as shown i 
Fig. 6. 
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Fig. 5—Channel graph. 


Fig. 6—Specified channel graph. 


It turns out that the balanced network in Fig. 6 can be constructed by 
a method in Ref. 6. However, that method is not able to construct the 
network in Example 1. 
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Automatic Numerical Quadrature 


By JAMES L. BLUE 
(Manuscript received April 15, 1977) 


An automatic numerical quadrature routine (ANQR) attempts to 


evaluate 
b 
f. fla) dx 


to absolute accuracy «, given only «, a, b, and a user-supplied subroutine 
which calculates f(x) for any x in [a,b]. An ANQR which guarantees 
success is impossible to construct, even disregarding the effects of finite 
computer precision, but the problem is nonetheless of interest. A reli- 
able and efficient ANQR is a necessary part of any mathematical sub- 
routine library. New single- and double-precision ANQRs, QUAD and 
DQUAD, have been constructed and tested. They are based on adaptive 
Romberg extrapolation, with cautious error estimation. An important 
practical feature is the automatic recognition of endpoint singularities, 
and a change of variable to handle them. QUAD and DQUAD also rec- 
ognize the presence of noise in the function being integrated, and limit 
the attempted accuracy accordingly. Since guaranteed ANQRs are 
impossible, extensive testing of DQUAD is presented to demonstrate 
its efficiency and robustness. Comparable testing is not available for 
competitive ANQRs, but performance ona standard set of test integrals 
is presented for DQUAD and nine other ANQRs. DQUAD Is generally 
better. QUAD and DQUAD are written in PFORT, a subset of American 
National Standard (ANS) Fortran. Machine-dependent constants are 
obtained from the PoRT library machine-constants programs. A por- 
table package of storage allocation routines is used. 


I. INTRODUCTION 


The development of automatic numerical quadrature routines 
(ANQRs) has been a popular research topic for many years (see refs. 1-6, 
8, 9, 11, 18, 14). An ANQR is a routine which attempts to calculate 


with absolute error, or perhaps relative error, no larger than «, given «, 
a, b, and a procedure which calculates f(x) for any desired x in the in- 
terval [a,b]. It is assumed that no other information about the function 
f is available. The problem is perhaps the more interesting for being an 
impossible one. Any numerical quadrature routine must estimate the 
integral by sampling the function f at a finite number of x’s. A guaran- 
teed automatic integration algorithm is clearly impossible for general 
f, even for analytic f. For example, given any deterministic rule for nu- 
merical quadrature, one can readily find constants a and 6 so that the 
quadrature rule calculates 


1 
Va f, e—a(x—B) dx 
0 


to be close to zero. (Choose a to be large and positive, and 8 to be between 
sampling points.) 

Although the general problem is impossible, one feels that an ANQR 
which works for “reasonable” functions should be feasible, and much 
work has been directed at this goal. There has been great confusion and 
difficulty in comparing the various candidates for ANQRs, partly because 
the domain of the problem is undefined; a reasonable definition of a 
“reasonable” function is itself difficult. 

In constructing an ANQR, an author is forced to make decisions about 
the class of “reasonable” functions, in effect to define what is a “‘rea- 
sonable” function. These decisions strongly affect the efficiency and 
robustness of the ANQR. For example, to avoid completely missing an 
isolated peak in f(x), the interval [a,b] must be sampled finely. However, 
a fine sampling is inefficient for easy functions. Another example is a 
function which is flat over 99 percent of the interval, and which has 200 
oscillations in the remaining 1 percent. If an ANQR is able to distinguish 
this function from one which is merely noisy over 1 percent of the in- 
terval, the ANQR is likely to be inefficient on easy functions and very 
inefficient on noisy functions. An ANQR which gives up relatively quickly 
on this function, calls it noisy, and returns an error message, may be 
preferable, especially since many such functions are the result of a user’s 
programing errors. 

A compromise strategy, used by QUAD, is to isolate all assumptions 
about the “reasonable” class of functions in a few parameters. Default 
values of these parameters can be chosen which will be suitable for most 
users. More knowledgeable users can use other values. With the default 
values, QUAD strikes what the author considers to be the proper balance 
between efficiency and robustness. 

Since no a priori information about f(x) is available, 


ff ax 
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must be evaluated by sampling f in [a,b]; the error in the calculated in- 
tegral is usually estimated by comparing two or more calculated values 
for the integral. ANQRs typically have a sequence of quadrature rules 
Q,, depending on a, b, and the function f, such that 


lim Q,, = f fix) dx 
n—o a 
if the calculations are done in infinite precision, and if f is at least 
piecewise continuous. Most ANQRs have no better error estimation 
procedure than to accept Q, whenever |Q,-1 — Qn| < «, a procedure 
fraught with danger. QUAD has a much more stringent error estimation 
procedure, described in Section II. 

Many functions to be integrated are easy to integrate over some parts 
of the interval and difficult over other parts. It is frequently more effi- 
cient to sample more densely in the difficult regions, if possible. ANQRs 
which attempt to do this are called adaptive—the points at which f is 
sampled depend on the function being sampled. An adaptive ANQR must 
include some strategy for how to concentrate the sampling points. Es- 
sentially all competitive ANQRs are adaptive. 

The usual adaptive procedure is to integrate an interval with quad- 
rature rules Q, forn =1,2,...,N, where N is fixed. Q, may be, for ex- 
ample, Simpson’s rule with 2N intervals, or Gauss-Legendre quadrature 
with n sampling points. If convergence has not been obtained, the in- 
terval is divided in half, and each half considered separately. For effi- 
ciency, one wants quadrature rules for which all sampling points for the 
whole interval are also used for the half-intervals. If the value of N used 
depends on the results Q,, for n < N, the method is sometimes called 
doubly-adaptive. 

Most ANQRs do not do well on integrals with endpoint singularities, 
but users’ integrals are frequently of this type. QUAD has a provision 
for recognizing endpoint singularities and for making a change of variable 
to facilitate the integration. This feature also works well on another 
important class of functions, those decaying steeply away from one or 
both ends of the integration interval. This automatic change of variable 
technique is a significant improvement over previous ANQRs. 

Most ANQRs cannot cope with noisy functions; if there is too much 
noise in f, most ANQRs fail in an unpleasant, uneconomical way. Con- 
vergence will be at best very slow, so that the ANQRs will stop only when 
their predefined limit on calls to the function evaluation procedure has 
been exceeded, with no indication that the problem is noise rather than 
a noise-free but unruly function f. QUAD recognizes noisy functions, sets 
a warning flag, and integrates only to an accuracy commensurate with 
the estimated noise. 

Finally, there is a large difference between an algorithm for numerical 
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quadrature and a properly-written ANQR suitable for a program library. 
Provision must be made, for example, to stop trying to integrate a 
function if it has been sampled more than some user-defined number 
of times. The finite machine precision of the computer involved must 
be taken into account. Temporary storage must not be allowed to over- 
flow. Provision for error returns must be made. 

The basic idea behind QUAD is adaptive Romberg extrapolation® 
combined with cautious error estimation’. The first such combination 
was the program CADRE, written by deBoor®. CADRE and QUAD are su- 
perficially similar, but differ in almost every detail. The major im- 
provements incorporated in QUAD include the following, which will be 
covered fully in Section II. 

(t) Noise. QUAD detects noisy functions and quits gracefully. 

(ii) Endpoint singularities. QUAD detects singularities in f(x) at the 
endpoints, a and b, and automatically makes a change of variable to 
reduce the strength of the singularity. 

(iit) Mesh sequence. QUAD uses the mesh sequence 1, 2, 3, 4, 6, 8, 12, 
16,..., instead of 1, 2, 4, 8, 16,..., giving a higher effective order of 
convergence. 

(iv) Portability. QUAD is written in PFORT,! a portable subset of ANS 
Fortran. Machine-dependent quantities are defined with the PORT’ 
machine constants. A portable Fortran stack’ is used for temporary 
storage. 

Section II discusses the algorithm of QUAD and DQUAD more fully. 
Section III compares the performance of DQUAD and nine competitive 
ANQRs on a standard set of test integrals, and also presents the results 
of some more serious testing of QUAD. Section IV discusses the imple- 
mentation of QUAD, including portability considerations. 


li. QUAD 
2.1 Romberg extrapolation 


QUAD is based on Romberg extrapolation of the composite trapezoidal 
rule.5 The formulas are standard, but will be repeated here for com- 
pleteness. Let n1, no, .. . be an increasing sequence of positive integers, 
and let h; = (b — a)/n;. Then the composite trapezoidal approximation 


to 
I= ff) dx 


(hi) = 5 hilfla) + f06)] + hi" fla + mh;) 


1654 THE BELL SYSTEM TECHNICAL JOURNAL, NOVEMBER 1977 


If f has 2k + 1 continuous derivatives in [a,b], then the Euler-Maclaurin 
sum formula shows that 


k 
T(h;) =IT+ > ¢mh?™ + O(hz*1) 
m=1 


where the c,, depend only on a, b, and f, not on h;. A higher-order, al- 
though not necessarily more accurate, estimate may be obtained by 
combining two trapezoidal estimates via Richardson extrapolation,? 
eliminating the c,h? term. Let T§ = T(h;). 


T® = TW 

T 1) 7 2) -_ ——_—_—_— 

h?/hz—-1 
=I + O(h}h}) 


Still higher-order estimates may be generated recursively. The general 
formula for generating Tt!) is 


; PD a Ty 
TW = Tt) 4 —#) = kel 
Po TEE + ky oI 
It is customary to think of the T-values as a table, viz. 

TY 
Te) fie 
TS) T\?) TW 
T\ T') TY) TY 
TO) TY TS) TY TY 


The classical Romberg method uses the sequence 1, 2, 4, 8, ... for the 
nj’s. 

Since at any time the right-most element, the “tip,” of the T table is 
of highest order, it is expected to be most accurate, and frequently is so. 
Early Romberg programs! tested for convergence only by checking 
successive tip elements of the table. There are several arguments against 
this practice. Firstly, for the tip there is no way to obtain an error esti- 
mate with any theoretical foundation. Secondly, highest-order is not the 
same as most accurate. Even for analytic functions, if the step size used 
at the beginning of the T table is too large, the tip of the table may not 
be the most accurate value. Thirdly, for functions f(x) which do not have 
enough derivatives, the tip of the T' table is not of higher order than the 
elements to the left. For example, if f(x) = x%, for 0 < a < 1, the low- 
est-order term in each column of the kth row is O(h@t!). In practice it 
is frequently found that lower-order columns are more accurate than 
the tip element. 
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2.2 Cautious error estimation 


The idea of cautious error estimation comes from Lynch.’ It is a simple 
and seemingly unobjectionable idea, but is not adopted by most authors 
of ANQRs. A cautious error estimation procedure believes an error esti- 
mate only if there is some evidence that the convergence rate of suc- 
cessive quadrature rules is close to the theoretical rate. Cautious error 
estimation is particularly easy for Romberg extrapolation. The two were 
first combined by deBoor.® QUAD’s version of catious error estimation 
is similar in spirit to deBoor’s, but is more cautious and is different in 
all details. 

It has been proven! that, if f is merely Riemann-integrable, each col- 
umn of the T table converges, as does each diagonal. If. f has enough 
continuous derivatives, 


Te =I+ Oh}... hha) 


These theoretical results provide a basis for cautious error estimation.®9 
Lynch’s suggestion was to consider three successive trapezoidal rule 
estimates, and form the ratios 


T® _ Tey 
TED am Thr?) 
Ze hj = hin + Ol(h? + hi?) 
hey _ hiss + O[(hit1 + h?+o)?| 


If the step sizes are small enough, the higher-order terms are small 
compared to the second-order terms, and 


RY = ns poh 
hai- hits 


RY = 


The calculated R§ being close to this theoretical value is good evidence 
that the convergence rate of the column is proper, and that the error in 
T® is dominated by c,;h?. Then the Runge estimate of the error,? 


Tet? — Te 
his Ship. — 1 


is likely to be a good estimate. If the calculated R{) is not close to the 
theoretical value, the Runge estimate of the error is likely to be an un- 
derestimate. 

Similar calculations are done in higher columns. The general formula 
for Ri is 


|Tht2 — 1 ~ = | TH?) — TED 








TP - THEY 


b) ot RS 
RP = Pe = Ty? 
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As all the h’s approach zero, 


2 2 2 
RY oe iss hi ies 


isk hist ma hi-nes 
The Runge estimate of the error in Tt”) is 


pir) _ Tey) 


uh i+2) _ I\ x 
IT | hapa o 1 


A more conservative estimate of the error in Tt), 
re = I| ~ | Tht?) = TE) 


is often used. 

QUAD calls column k asymptotic if R{? is close enough to the theo- 
retical value; the tolerance is 5 percent of the theoretical value for k = 
0, 10 percent for k = 1, 15 percent for k = 2, and so on. Column k is al- 
most asymptotic if R{? is between 0.25 and 4.0 times the theoretical 
value, except for column 0, where the criteria are 0.75 and 1.25. 

If columns 0 through k are asymptotic, QUAD believes the Runge es- 
timate for the error in the kth column. If columns 0 through k — 1 are 
asymptotic and column k is almost asymptotic, QUAD believes the 
conservative estimate for the error in the kth column, but no higher 
columns are believed. (The only exception is that, if the column 0 is only 
almost asymptotic, the next column is believed if it itself is asymptot- 
ic.) 

This describes the basic cautious error estimation procedure for QUAD. 
There are a few more details, however. QUAD does not believe any answer 
based on less than two extrapolations, or five sampling points, per in- 
terval. If two successive entries in a column give the same value to within 
a few rounding errors, as occurs when integrating a constant function 
or in doing very accurate integration, then the column does not appear 
to be asymptotic. The conservative error estimate for the column is be- 
lieved anyway. If an interval has a singularity, either real or due to 
rounding errors or truncation errors in the function subprogram for f(x), 
no column will appear asymptotic. Then a nonasymptotic answer in the 
first column will be accepted after several extrapolations, with the very 
conservative error estimate 


| Thr?) _ I| ww 2|T§t?) 2. TH) ae oT ey = T$| 


Finally, at any stage one column of the T table has only two entries in 
it, and cannot be judged to be asymptotic or nonasymptotic. The con- 
servative error estimate is accepted for such a column if the previous 
column is asymptotic. 
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2.3 Step size sequence 


The above discussion applies to any sequence of step sizes. The clas- 
sical Romberg sequence uses step sizes (b — a)/n, with n = 1, 2, 4, 8, 16, 
..., halving the step size and doubling the number of sampling points 
at each new extrapolation. Several alternative sequences have been 
suggested which do not cause the number of sampling points to rise so 
rapidly. QUAD uses n’s of 1, 2, 3, 4, 6, 8, 12,. . , another reasonable pos- 
sibility is 1, 2, 3, 4, 5, 6, 8, 10, 12,.... These sequences double the number 
of sampling points every second and third extrapolation, respectively. 
The classical sequence has the advantages that the bookkeeping is very 
easy and that all old sampling points are reused if the interval is divided 
in half. The latter is essential for efficiency. 

QUAD’s sequence uses fewer sampling points to get the same accuracy, 
as suggested by Bulirsch and Stoer.? The bookkeeping is more compli- 
cated than for the classical sequence. For example, it is only convenient 
to divide the interval in half after the fourth, sixth, eight, ... extrapo- 
lations if all the old sampling points are to be reused. 


2.4 Adaptive procedure 


For an adaptive Romberg extrapolation routine, it is necessary to 
decide when to do another trapezoidal rule and another extrapolation, 
and when to divide the interval. The minimum number of extrapolations 
for QUAD is 4, using step sizes (b — a) down through (b — a)/6, and a 
total of 9 sampling points. This default lower limit may be raised by the 
user (see Section IV). Because of roundoff, unlimited extrapolations are 
impractical—the highest-order columns will not be asymptotic and will 
not be believed. The maximum number of extrapolations allowed by 
QUAD is 6; DQUAD allows 8. The default limit may be changed by the user 
(see Section IV). 

If the requested error tolerance for an interval has not been achieved 
after 4 extrapolations, QUAD goes on to 5 and 6 extrapolations if the first 
column is asymptotic; after 6, it goes on if the second column is also as- 
ymptotic. This procedure is biased in favor of doing more extrapolations, 
and trying to get higher-order convergence, for smooth functions. 
Functions which are not smooth, or which do not appear asymptotic 
because of too large a step size, have the interval divided instead of 
having more extrapolations done. If QUAD decides to divide an interval, 
the lower half is stacked, and the upper half is attempted next. 


2.5 Change of variable 


Functions with singularities are expensive to integrate without special 
methods. Since the interval containing the singularity will have a 
nonasymptotic T table, convergence will be limited to the first column. 
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For example, it can be shown!” that for 


1 
[= f, f(x) dx = f, xg(x) dx (1) 
0 
where g is smooth, and f(0) is set equal to zero, 
Tih) =I+ 3. ¢mh2™+ Yo dahmte (2) 
m=1 m=1 


The dominant error term for the first column of the T table is likely to 
be the d,h!*¢ term, so convergence is slow. If f(0) is not zero, another 
infinite sum is added to (2), like the second sum, but with a = 0. 

For such an endpoint singularity, the error is of a simple form. It is 
feasible to recognize this type of singularity in the same way that the 
cautious error estimation procedure recognizes asymptotic, or h?™ be- 
havior. De Boor® does exactly this, estimates an a, and then extrapolates 
using eq. (2). The success of this procedure depends critically on how 
accurately a can be estimated. If f(0) is not set equal to zero, de Boor’s 
method will not work well. For logarithmic singularities, the error ex- 
pansion corresponding to eq. (2) is more complicated; de Boor makes 
no attempt to recognize logarithmic singularities. 

After recognizing an endpoint singularity, QUAD uses a different 
procedure. Suppose that the integral is as above, where g is well-behaved. 
Then the leading error term is O(h!*°) if -1 <a < 1. In the second and 
higher columns, the O(h2) term is gone, so the leading term is O(h!*2) 
for —1 < a < 3. QUAD looks at ratios of T table entries in the second and 
third columns to recognize x“ behavior, and estimates the value of a. 
QUAD then makes a change of variable x = uw”, where rn is the closest 
integer to 6/(1 + q), giving for eq. (1) 


1 1 
f, nu"—1f(u”) du= {i nunta)-Is(y4") du 


(The change of variable is somewhat more complicated if the limits of 
integration are not 0 and 1.) The new integral has a singularity of the 
form u®, where @ is between 4.5 and 5.5, if @ is close to the true a. The 
singularity in the transformed integral is lessened, allowing convergence 
in the second or third columns of the T table. Convergence is likely to 
be much quicker. For rapid convergence, the method does not rely on 
the estimated @ being close to the true a or upon eq. (2) holding, or indeed 
on there being any singularity at all at the endpoint. 
Steeply decaying integrands such as 


1 
100 §, e100 dy 
0 
look like step functions when coarsely sampled. A step function is an x° 
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singularity, since (0) = 0, so a change of variable is made with n = 6. 
Singularities of the form x log (x) will look sufficiently like x% singu- 
larities for some a, so that the transformation will be made. (It is not 
obvious that this is true, but tests have strongly indicated that it is.) 

If ais close to —1, n can become large. QUAD requires n to be less than 
a maximum value determined by the precision of the computer; this 
value was 22 for tests reported in Section III. The change of variable is 
not made if @ is less than —0.99. 

To facilitate the change of variable, QUAD starts by dividing the in- 
terval [a,b] into three equal intervals, and reverses the upper third. 
(Three is the default number, and may be changed by the user—see 
Section IV.) On the lower third and the reversed upper third, a left-hand 
endpoint singularity is recognized by a pattern of “fail on whole interval, 
succeed on right half-interval” twice in a row, and is followed by the 
estimation of a. If the two estimated values for a from the second and 
third columns of the T table do not agree to within 0.1, no change of 
variable is made. No change of variable is attempted except at the two 
endpoints of the originial interval. 


2.6 Nolsy functions 


All procedures for evaluating f(x) are inherently noisy, since they are 
implemented on finite-precision machines. The value returned is not 
the exact f(x), but f(x) [1 + r1(x)] + re(x), where r;(x) and ro(x) are noise 
functions. Ideally, r; could always be no larger than a few rounding errors, 
and rz could be no larger than a few times the smallest positive ma- 
chine-representable number. Noise of this size should not affect the 
performance on an ANQR unless ¢ is very small, of the order of ro(x) or 
f(x)ry(x). 

Protecting against this magnitude of noise is quite easy, although few 
ANQRs bother to do so. QUAD estimates a priori the sizes of r; and ro, 
based on the machine precision, and requires all error tolerances to be 
at least as large as the estimated rounding error. 

Protecting against significantly larger noise is more difficult. A suc- 
cessful ANQR should recognize the presence of noise, estimate its mag- 
nitude, and evaluate the integral in a “reasonable” number of function 
evaluations with an accuracy which is “nearly” as good as possible. 
(There is of course a trade-off between “reasonable” and “nearly.’’) If 
typical values of ro(x) or of f(x)r1(x) are much larger than ¢/|b — a|, most 
ANQRs will fail in an unpleasant, uneconomical way. Convergence will 
at best be very slow, so that the ANQRs will stop only when their prede- 
fined limit on calls to the function evaluation procedure has been ex- 
ceeded, with no indication that the problem is noise rather than a 
noise-free but unruly function f. 


1660 THE BELL SYSTEM TECHNICAL JOURNAL, NOVEMBER 1977 


SQUANK!! makes a reasonable effort at recognizing noise. However, 
it attributes any nonstandard behavior to noise, so that some unruly but 
noise-free functions are called noisy. 

Qualitatively, one may say that a function is noisy if, on a “sufficiently 
small” interval, the values of the samples of f are “not smooth enough.” 
An algorithm consists of the defining of “sufficiently small” and “not 
smooth enough,” followed by estimation of the magnitude of the noise 
and by further action to avoid using an excessive number of function 
values. 

In QUAD, no answer is believed unless the function has been sampled 
with adjacent samples no farther apart than h,|b — a|; h, is supposedly 
small enough so that all structure may be seen by sampling with this 
spacing (h, is parameter HSAMPL of Section IV). The default value of 
hg is 4. Noise is not estimated unless adjacent samples are no farther 
apart than h,,|b — a|, with h, = h,/32. Choosing h, smaller would re- 
quire more function values; choosing h,, larger would increase the risk 
of calling a function noisy when it is noise-free but rapidly varying. 

When a function is noisy, QUAD will usually fail on large intervals, and 
then attempt smaller and smaller subintervals. When integration on a 
subinterval has failed, and adjacent samples are no farther apart than 
h,,|b — al, the noise in f is estimated. First, the second differences of the 
samples of f on the subinterval are formed, e.g. f(x) — 2f(x + h) + f(x 
+ 2h). If the sequence of second differences has no more than two sign 
changes, noise is not assumed to be present. If there are three or more 
sign changes, noise is assumed to be present, since the function has too 
much structure over too small an interval, and the estimated answer and 
error for that subinterval are accepted as being as good as possible. 

When noise has thus been found to be present, the second differences 
are assumed to be essentially all noise, and the magnitude of the noise 
is estimated as the average of the absolute values of the second differ- 
ences. All succeeding subintervals are attempted with accuracy not ex- 
ceeding the estimated noise magnitude times the length of the subin- 
terval. If other subintervals are found to be noisy, the largest noise 
magnitude is used. 


2.7 Error allocation 


QUAD attempts to integrate the upper one-third of [a,b] with error 
tolerance </3. Then the following procedure is used to assign an at- 
tempted accuracy for each interval. When integration on an interval is 
attempted with error tolerance ¢9 and fails, the upper half is attempted 
with error tolerance ¢9/2. When integration on an interval succeeds, the 
absolute value of the estimated error is added to a running sum, and the 
top interval is the stack is attempted. If the top interval is of length 6x, 
the total length of intervals remaining to be integrated is x1, and the sum 
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of the absolute values of the error estimates so far is «1, then the error 
tolerance assigned to the top interval in the stack is (€ — €,) 6x/x. 
However, the error assigned to any interval is required to be at least as 
large as «/1000. 

If an answer is returned for an interval with an error estimate which 
is larger than the requested error, but less than the estimated roundoff 
or noise, that answer and error are accepted, and a warning flag is set. 


lil. TESTING AND COMPARISON OF ROUTINES 


Testing is necessary to evaluate the efficiency and robustness of an 
ANQR. Typically the proposer of an ANQR generates an algorithm, codes 
a simple program, and tests the routine on a few integrals of the pro- 
poser’s own choosing. It is not unusual for all the test integrals to be done 
well by the ANQR. As a reqult, the prospective user has no way of eval- 
uating the quality of the ANQR without performing extensive testing. 

Some improvement was evidenced in the work of Kahaner,® who tested 
many ANQRs on the same set of 21 test integrals. The same set was used 
by de Boor® for testing his ANQR. At least three of the 21 are not appro- 
priate test integrals, however, because the results are algorithm-de- 
pendent in an unrepresentative way. 

Two examples will make this clear. First consider 


§, "Se cou (an ae 


where f(x) is any smooth function. QUAD, which divides according to the 
1, %, Ys, %4, Ys sequence, will fail for a near 36, but not for a near 32. [For 
a near 36, the regular sampling procedure of QUAD samples only near 
the peaks of the cosine, so the integrand looks like f(x).| CADRE,® which 
divides according to the 1, 14, 1/4, 1% sequence, will fail for a near 32, but 
not for a near 36. A single test integral with a large a may not compare 
ANQRs fairly. Test integrals 13 and 17 of Kahaner (see Table I) are of 
this type, each having about 50 full cycles. 
Second, consider 


1 
Va f, e—a(x—B)? dy 
0 


for a large and positive. Depending on the choice of 8, this can be either 
easy or hard for a particular ANQR. If the peak comes sufficiently near 
a sampling point, adaptive ANQRs can zero in on the peak and integrate 
it accurately, although many sampling points will be necessary. If the 
peak does not come sufficiently near a sampling point, the integrand 
looks like zero, and the ANQR fails. For proper comparison of ANQRs, any 
single a is insufficient. Test integral 21 of Kahaner is of this type. 
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Table | — Kahaner’s 21 Test Integrals 


No. a b answer f(x) 
Easy 
12 0 1 +0.7775046341 x/(e* — 1) 
11 0 1 +0.3798854930 1/(1 + e*) 
1 0 1 +1.7182818284 e* 
10 0 1 +0.6931471806 1/(1 + x) 
4 -1 1 +0.4794282267 0.92 cosh (x) — cos (x) 
8 0 1 +0.8669729873 1/(1 + x4) 
5 -1 1 +1.5822329637 1/(x4 + x? + 0.9) 
20 -1 it +1.5643964441 1/(x? + 1.005) 
Steeply 
decaying 
15 0 10 +1.0000000000 25e ~25x 
14 0 10 +0.5000002112 V/50 exp (—507x2) 
16 0 10 +0.4993638029 50/[p(1 + 2500x?)] 
Singular 
6 0 1 +0.4000000000 x3/2 
2 0 1 +0.6666666667 x1/2 
2 0 1 +0.7000000000 0,x < 0.3; 1, x > 0.3 
7 0 1 +2.0000000000 0,x =0;x71/2,7% >0 
19 0 1 —1.0000000000 0,x = 0; In (x),x >0 
Oscillatory 
18 0 P +0.8386763234 cos [cos x + 3 sin x + 2 cos 2x + 
3 sin 2x + 3 cos 3x] 
9 0 1 +0.4794282267 A + sin (10px)] 
17 0.01 1 +0.1121395696 50[sin (50px)/(50px)]? 
13 0.1 1 +0.0090986453 sin (100px)/px 
Isolated peak 
Zz 0 if +0.2108027354 sech?[10(x —0.2)] + sech*[100(x — 


0.4)] + sech®[1000(x — 0.6)] 
Note: p = 3.14159, P = 3.1415927. 


Kahaner did not test noisy functions, and did not ask for impossibly 
small error tolerances. 


3.1 Testing on the Kahaner 21 


The Kahaner 21 are listed in Table I. They have been grouped ac- 
cording to type; within groups they are ordered approximately by dif- 
ficulty. Tables II, III, and IV summarize the results of ten ANQRs on the 
21 test integrals, for requested error tolerances 10-3, 10-6, and 10-9. In 
each table, the first column is the integral number; succeeding columns 
are the number of sampling points used by each ANQR. An F indicates 
that the ANQR was unsuccessful, and an asterisk that the ANQR used 
the fewest sampling points of any successful routine. (For these tables 
only, “successful” means that the true error of an integration is no more 
than 20% higher than the requested error, since some ANQRs failed by 
a small amount.) Columns labeled ROMB through RBUN are based on 
the number of sampling points reported by Kahaner® for seven of his 
highest-quality ANORs, and were obtained on a CDC 6600. Column 
CADRE is from de Boor,® and the integrals were performed on a CDC 6500. 
Columns QSUBA and DQUAD were computed especially for this com- 
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Table Il — Number of sampling points used by each ANQR; 
attempted absolute accuracy 107° 


Inte- 
gral ROMB SIMPSN SQUANK QNC7 QNC10 QABS RBUN CADRE QSUBA DQUAD 


12 17 19 25 3718 5* 9 


9 7 
ll 17 19 9 25 37 13 5* 5* 7 13 
1 17 19 9 25 37 13 5* 9 7 13 
10 17 19 9 25 37 13 5* 9 vi 13 
4 17 19 9 25 37 13 5* 17 7 13 
8 17 19 9 25 37 13 5* 9 7 15 
5 17 19 9* 25 37 13 11 33 15 17 
2 7 3 9 2 37 18 IW 17 #15 19 
15 513 103 53 85 109 85 527 88 63 52* 
14 1025 103 49* 97 127 85 51 62 3F 52 
16 2049 115 53 121 163 109 87 81 127 48* 
6 17 19 9 25 37 13 5* 9 7 15 
3 65 55 oF 49 55 77 211 17 15* 17 
2 257F 115 29F 121 163 141 271 53* 771 85 
7 81938F 235 105F 241 861 1838F 211 33 517 26* 
19 4097 175 45 217 307 181 211 137 31 28* 
18 129 139 53* 85 73 17 39F 107 63 61 
9 33* 163 81 97 145 149 719 183 127 101 
17 1025 151 57* 165 307 149 109 512 255 185 
13 1025 19F 429 49F 865 573 533 1028 255* 381 
21 4097 127 17F 97 127 77 65 108 333F 49* 
* = best of all successful results 
F = failure 


parison, on a Honeywell 6070. Double precision was used so that the 
relative machine precision would be comparable to that of the CDC 
machines. QSUBA!‘ has provision only for relative error; for these tests 
a relative error was requested which gave the appropriate absolute error 
request. 

It is important to notice that some of the failures are due to an ANQR 
deciding to stop because of excessive sampling; these failures are far less 
reprehensible than the others because an error return could have been 
made, and an incorrect answer rejected. For RBUN through ROMB, this 
information can only be inferred since Kahaner did not list any error 
returns. No such failure occurred for CADRE or DQUAD; QSUBA has no 
built-in maximum. 

ROMB! is astandard Romberg extrapolation routine, using the stan- 
dard 1, 4, 14, 4g sequence. It is not adaptive. It stops, apparently, after 
8193 sampling points of f(x). Thus only one of its failures is serious. 
ROMB requires at least 17 points before believing any answer. 

SIMPSN® and SQUANK!! are adaptive Simpson’s rule routines, ap- 
parently with cutoffs at 5003 and 5001 points, respectively. These rou- 
tines are decent at low accuracies, but are not of high-enough order to 
be competitive at high accuracy. SQUANK also assumes that an improper 
convergence rate is due to noise in the function, rather than to a singu- 
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Table Ill — Number of sampling points used by each ANQR; 
attempted absolute accuracy 1076 


Inte- 

gral ROMB SIMPSN SQUANK QNC7 QNC10 QABS RBUN CADRE QSUBA DQUAD 
12 17 19 9 25 37 13 5* 9 7 13 
11 17 19 9 25 37 13 Il 9 7* 13 
1 17 55 17 25 37 13 21 17 qT* 19 
10 17 55 21 25 37 13* 31 17 15 19 
4 17 55 25 25 37 25 5F 33 15* 19 
8 33 67 29 25 37 25 Al 17 15* 21 
5 65 163 65 49 37 49 59 49 31* 41 
20 65 163 49 49 37 49 49 33 31* 33 
15 2049 343 213 133 145 133 4117 140 63* 98 
14 2049 331 169 133 163 109 91 89 3F 86* 
16 81938F 511 273 181 181 145 141 145 255 96* 
6 129 91 29* 61 73 65 383 65 31 34 
3 4097 199 105 157 217 145 423 33 63 32* 
2 81938F 235 29F 241 361 261 271 119* 3351 125 


7 §81938F 1027F 1153F 241F 361F 89F 587F 129 5925 40* 
19 8193F 499 257 241 361 105F 403 233 795 38* 


18 = 257 547 301 181 199 =—-205-—Ss:«195 177 63* 117 

9 129* 871 377 289. 397 313 267F 409 255 = 285 
17 2049-2275 697F 385F 1009 829 697 1237 255* 547 
13 2049 19F 2549 1525 1639 1449 2383 1449 255* 757 


21 81938F 691 185F 205F 253F 197F 327* 189F 525F 127F 


* = best of all successful results 
F = failure 


larity, and so quits early on some of the test integrals. SIMPSN requires 
at least 19 points, and SQUANK 9. 

QNC7 and QNC108 are adaptive Newton-Cotes routines, with 7- and 
10-point rules, respectively. (QNC10 was called QUAD in Ref. 8.) They 
performed quite well, failing only on some of the most difficult integrals. 
QNC7 requires at least 25 points, and QNC10 at least 37, somewhat ex- 
cessive for the easiest integrals. 

QABS!8 combines Romberg and Curtis-Clenshaw quadrature, and 
performed quite well, failing only on some of the most difficult integrals. 
It requires at least 13 points. 

RBUN? is an adaptive Romberg extrapolation routine, using the 
standard sequence. It apparently has a cutoff at 5001 points. RBUN re- 
quires at least 5 points, and seems to be somewhat unreliable. 

Kahaner recommended any of QNC7, QNC10, or QABS as a library 
routine. He did not have CADRE, QSUBA, or DQUAD available to test. 

CADRE® is more recent than the ANQRs just discussed. It uses a version 
of cautious Romberg extrapolation based on the standard sequence. It 
also includes provision for recognizing a singularity of the form x°, es- 
timates a numerically, and extrapolates using the estimated a. CADRE 
requires at least 5 points. On the test integrals, it seems somewhat more 
efficient than QNC7, QNC10, and QABS on nonsingular integrals and 
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Table IV — Number of sampling points used by each ANQR; 
attempted absolute accuracy 107° 


Inte- 

gral ROMB SIMPSN SQUANK QNC7 QNC10 QABS RBUN CADRE QSUBA DQUAD 
12 17 55 33 25 37 13* 39 17 15 19 
11 33 151 33 25 37 25 39 17 15* 19 
1 17 163 65 25 37 25 73 17 15* 23 
10 65 271 97 37 37 49 123 33 15* 25 
4 33 331 105 25 37 85 139 33 15* 25 
8 65 463 149 73 73 97 129 65 31* 45 
5 =129 487 289 97 73 181 239 129 31* 73 
20 129 487 249 97 73 145 185 129 31* 49 
15 4097 1483 1145 241 217 281 5001F 215 127* 1388 
14 4097 1123 7197 241 2538 245 259 202 3F 154* 
16 8193F 2467 1649 897 343 397 1485 337 255 192* 
6 2049 427 161 133 163 137 1423 529 63 46* 
3 81938F 883 513 289 ~=—s- 36:1 289 1595 129 255 42* 
2 8193F 235 29F 241F 361F 381 271 173 5931 165* 


7 81938F 4279F 5001F 589F 685F 89F 2467F 625 11325 70* 
19 81938F 2203 1969F 421F 415F 89F 1571 369F 3495 80* 


18 513 2923 1589 409 343 589 753 417 127* 217 
9 257* 3967 2525 697 157 893 883 785 765 473 
17 4097 5008F 5001F 1845F 1999 2025 2741 23829 255* 1109 
13 4097 50038F 5001F 3073 27738 3197 5001F 3505 255* 1161 
21 8198F 3751 1657 709 685 633* 1079 661 827F 261F 
* = best of all successful results 
F = failure 


much more efficient on singular ones. In addition, the cautious extrap- 
olation means that CADRE’s error estimation procedure has some ra- 
. tionale behind it, and CADRE is more robust than the aforementioned 
routines. However, CADRE is difficult to understand and to maintain, 
since its style is the antithesis of structured programming. It is one large 
program, with no subprograms, but with a liberal and unstructured use 
of GOTOs. 

QSUBA/ uses a series of 8 whole-interval quadrature rules of increasing 
order, starting with the 1- and 3-point Gauss-Legendre rules. Succeeding 
rules are constructed to be of as high order as possible, consistent with 
using all the previous sampling points. The highest-order rule uses 255 
points and is of order 383. If convergence is not obtained after 8 rules, 
the interval is divided in half, and each half considered anew. Unlike all 
the other ANQRs under consideration, all function values must be dis- 
carded, since none of the sampling points on the half intervals coincides 
with any on the full interval. QSUBA works well on any integral which 
can be integrated without dividing the interval, and poorly on integrals 
which require dividing. It is especially good on easy and oscillatory in- 
tegrals, since a high-order rule is generally used. QSUBA uses at least 3 
points, which is somewhat unsafe, but has no maximum built in—it goes 
on forever, if necessary. 
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DQUAD takes at least 13 points. It fails only on number 21, for high 
accuracy, missing the narrow peak at x = 0.6. The integral which is the 
same as 21 except for moving the peak to 0.61 is done properly, using 117, 
241, and 447 points for error tolerances 10—°, 10-6, and 10-9, respectively. 
DQUAD is clearly more efficient and robust, based on these test integrals, 
than any other ANQR tested except QSUBA. DQUAD is more robust than 
QSUBA, but is less efficient for integrals where QSUBS does not need to 
divide the interval. 


3.2 Parameter studies (1) 


Testing an ANQR on a “random” set of test integrals, while instructive 
and a good start, is insufficient for a library routine. The testing of an 
ANQR is incomplete without numerous parameter studies: 


ff fee) dx 


with fixed ¢ and varying a (Ref. 12), and with fixed a and varying «. The 
function f(x,a@) should be increasingly difficult to integrate as a ap- 
proaches some limit, and a should be pushed close enough to that limit 
so that failure occurs. For error tolerance studies, the requested error 
should range from the approximate value of the integral to less than 
typical roundoff on the computer being used. 

Several of the first type of parameter study will now be discussed. For 
all of them, the error requested is 10~®. The first was suggested by de 


Boor®: 
1 Da 
———d 
i) 1+ (2%x)2 


For large a, the integrand is highly peaked. The integrand is 2% at x = 
0, falls to half that at x = 2-¢, and is 2—“ at the endpoints of the interval. 
There is no danger in missing the narrow spike, since it is exactly at the 
center of the interval, a normal sampling point for almost all ANQRs. This 
example demonstrates the power of adaptive ANQR, in that the number 
of sampling points increases only as Va , approximately. The behavior 
of DQUAD is shown in Fig. 1. The number of sampling points is plotted 
against a. The meaning of the symbols used for plotting in all the figures 
is given below. 


Successful integration; no error flag 
xX Unsuccessful integration; error flag 
O Unsuccessful integration; no error flag 
® Successful integration; error flag 


In testing DQUAD, “successful” means that the true error is less than e. 
DQUAD failed for a > 38, but recognized its failure (failure was due to 
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Fig. 1—Performance of DQUAD on function highly peaked at center of interval. 


filling of the function value stack). For comparison, CADRE fails starting 
at a = 31, and generally uses about 20 percent fewer sampling points than 
DQUAD. 

This integral should not pose any serious difficulty to a decent ANQR, 
since the unpleasant behavior occurred exactly at the center of the in- 
terval. Usually, though, the user should strive to break up integrals so 
that any unpleasant behavior happens at one of the endpoints of the 
interval. It is feasible for ANQRs to recognize such behavior at the end- 
points, but difficult if it occurs in the center of the interval. As an ex- 
ample, the previous integral, except with limits 0 and 1, may be consid- 
ered. DQUAD’s performance is shown in Fig. 2. For a = 7, DQUAD recog- 
nizes that the integrand approximates a step function, and makes a 
change of variable. This change of variable keeps the number of sampling 
points from growing significantly as a increases. 

Figure 3 shows a similar integral, except more steeply decaying away 


from the endpoint. 
1 
§, 2ae—2%% dx 
0 
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Fig. 2—DQUAD’s performance on function highly peaked at end of interval. 
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Fig. 3—Performance of DQUAD on steeply decaying function. 


Failure eventually occurs, as for the previous integral, when, after the 
change of variable, the new integrand is a sharply-peaked function with 
its peak away from an endpoint, and the peak is missed entirely. 


Testing of ANQRs on functions with isolated peaks within the range 
of integration takes more work; a must parameterize the narrowness of 
the peak, but the position of the peak is also important. A suitable test 
integral has two parameters. DQUAD was tested on 


1 
f, Qae—4(x—8)? dy 
0 


For each a, 25 integrals were done, with 6’s of 0.02(0.02.)0.50, for a sta- 
tistical evaluation. No failures occurred for a = 1, 2, 3, 4, 5, or 6; one oc- 
curred for a = 7, at 3 = 0.04. For a = 8, 12 out of 25 failed. Figure 4 il- 
lustrates the results for 8 = 0.40, a typical value. 

Another standard test integral, also used by de Boor, is 


a 
f, x*dx 
0 


1 Z 
2 
| 2a ,-4% (x-0.4)2 4. 
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Fig. 4—Performance of DQUAD on highly peaked function. 
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Fig. 5—Performance of DQUAD on function with endpoint singularity. 


with the integrand set equal to zero at x = 0. Thus for a = 0 the integrand 
is a step function, and for a < 0 the integrand has an infinite disconti- 
nuity. For a < 0, the integrand is “unreasonable” by almost anyone’s 
definition, but users sometimes give such integrals to ANQRs. Figure 5 
shows the performance of DQUAD. A change of variable was made au- 
tomatically by DQUAD for —0.97 < a < 1.75, approximately. DQUAD is 
designed to reject the change of variable if the estimated a is less than 
—0.99, and the change is not necessary to achieve the desired accuracy 
for a 2 1.75. For a < —0.97, DQUAD fails with an error flag, using about 
900 function samples. Machine precision limits the efficiency of the 
change of variable for a less than about —*4. For comparison, CADRE 
fails, with an error flag, for a less than —‘%f, and gives an erroneous error 
flag for a near, but not at, a = 1. CADRE is substantially less efficient than 
DQUAD for this test integral. Other ANQRs, without special procedures 
for endpoint singularities, are much worse. 

A final type of test integral is one with an oscillatory integrand. Since 
almost all ANQRs sample the integrand at regularly spaced points, there 
is a danger of undersampling. As an example, consider 


f [1 + cos (axx)] dx 
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Fig. 6—Performance of DQUAD with oscillatory integrand. 


as suggested by de Boor. Figure 6 illustrates the performance of DQUAD 
on this test integral. The first failure is for a near 36, where the regular 
sampling causes the cosine to look like unity. The lower part of the figure 
shows the region near 36. The top part shows the number of sampling 
points used for a from 1, to 20, with spacing of 1. The trend line is ap- 
proximately N = 10a; CADRE’s trend line is approximately N = 20a. 
Besides the general rising trend, there many dots significantly below the 
trend line. These occur because of resonance between the regular sam- 
pling of DQUAD and the regular oscillation of the integrand. If a is inte- 
gral, or very nearly so, coarse sampling may indicate that the integrand 
is simpler than it really is. For example, if a is an odd integer, the cosine 
is odd about x = 0.5, and the center third of the integral is integrated 
(correctly) based on insufficient sampling, since the trapezodial rule 
correctly integrates odd functions. 

This kind of resonance phenomenon is a difficult problem for an ANQR 
to surmount. The best way is probably to back off slightly from the no- 
tion of a fully automatic ANQR. Usually a prospective user of an ANQR 
knows if an integrand is oscillatory, and further, knows the approximate 
period of the oscillation. To avoid problems with possible undersampling, 
the user could divide the integral into several integrals, each with only 
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Fig. 7—-Parameter test with accuracy varied. 


a few periods of the oscillation. An easier way is for the user to require 
the ANQR to sample the integrand sufficiently finely to see all the os- 
cillations; this assumes the user knows the approximate period. DQUAD 
has provision for this, using the parameter HSAMPL, discussed in Section 
IV. If HSAMPL is taken to be 1/a in the above example, all the dots fall 
on the main trend line, and no failures occur. 


3.3 Parameter studies (2) 


The second type of parameter testing, using a single test integral and 
varying the requested error, will be considered now. Some ANQRs will 
fail when the requested error is large compared to the value of the inte- 
gral, because of insufficiently cautious error estimation. All ANQRs fail 
if the requested error tolerance is too small, because of roundoff. A 
properly designed library ANQR will have some mechanism for dealing 
with too-small error tolerances. Finally, the graph of the number of 
sampling points versus the requested accuracy is of interest. 


Figure 7, for 
1 
J, 8e~8 dx 
0 


is typical of DQUAD’s performance on easy integrals. The graph of N 
against log 1/e is horizontal at large requested error, with no failures, since 
DQUAD does not believe any answer until it can be confident of the error 
estimate. The central portion of the graph is roughly linear. The number 
of sampling points needed is approximately proportional to «71/20, 
DQUAD is effectively functioning as a 20th-order method. The small- 
est-error part of the graph is also horizontal; accuracy is limited by 


roundoff. 
1 
f, ly 
0 


Figure 8, for 
is similar except for the smallest requested errors. A change of variable 
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Fig. 8—Parameter test with accuracy varied. 


is made only for « = 10~4 through 107!6, For « = 10~? through 10-®, 

DQUAD is effectively a 15th-order method. The effective order is less than 

for the previous example because only three columns of the T table can 

be asymptotic. Roundoff begins to pollute the answer at « = 10~16, 
Figure 9, for 


f [1 + cos (arx)] dx 


for two values of a, 1.95 and 17.95, is the last example. For a = 1.95, 
DQUAD is effectively a 20th-order method for ¢ = 1073 through 107!". 
For a = 17.95, DQUAD does not begin to be a 20th-order method until 
e= 1076. 


3.4 Parameter studies (3) 


The final type of parameter testing uses a single test integral and error 
tolerance, but varies the amount of noise in the function. For testing, 
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Fig. 9 Parameter test with accuracy varied. 
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Fig. 10—Parameter test with amount of noise varied. 


integrals of the types 


f [f(x) + 10¢r(x)] dx 


f, : f(x)[1 + 10¢r(x)] dx 


were done, with requested accuracy « = 10-6. The function r(x) is the 
output of a pseudorandom number generator with values uniformly 
distributed on (—1,1). Values used for a were 1, 0, -1,...,-8. For f, the 
same four functions were used as in the previous section. For each 
function and each a, five different starting points of the random number 
generator were used. Sample results are shown in Fig. 10 and 11. The 
number of function values used is plotted vs. a, for each of the five tries. 
(Where two or more of the tries coincide, they are plotted side by side.) 
Points plotted with - are those for which DQUAD returned with an error 
estimate less than 10-6; points plotted with X indicate an error estimate 
greater than 10-6. 
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Fig. 11—Parameter test with amount of noise varied. 
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A summary of the results follows. 

For every test with 10% > 10-®, DQUAD recognized the presence of 
noise, estimated its magnitude, adjusted accuracy tolerances accordingly, 
and gave an answer and error estimate. The error estimates given were 
at most twice 10%, and were generally less than 1.2 times 10%. In only 6 
out of 280 integrals was the calculated answer farther from the noise-free 
value of the integral than the estimated error returned. 

For every test with 10% = 10-6, DQUAD returned an answer with an 
error estimate less than 10~°. In 3 out of 40 integrals, the presence of 
noise was recognized. In 2 out of 40, the calculated answer was between 
1 and 2 times 10-6 away from the noise-free value; in the remaining 38, 
the answer was less than 10~§ away. 

For every test with 10% < 10-6, DQUAD returned an answer with an 
error estimate of less than 10-6, and the calculated answer was less than 
10~6 away from the noise-free value. No noise was recognized. 


IV. IMPLEMENTATION OF QUAD 
QUAD is a Fortran subroutine which attempts to evaluate 


ff fe) ax 


to within absolute accuracy «; the user supplies a, 6, «, and a Fortran 
function subprogram to evaluate f(x). The discussion also applies to the 
double precision version, DQUAD, except as noted. 

QUAD is written in PFORT,! a portable subset of ANS Fortran. Tem- 
porary storage space is managed by a portable Fortran stack mecha- 
nism.” 

The calling sequence is 


CALL QUAD (F,A,B,EPS,ANS,ERREST) 


F is the name of the user’s subprogram, A and B are the limits, and EPS 
the requested accuracy. The estimated value of the integral and an es- 
timate of the accuracy of the estimate are given in ANS and ERREST. If 
ERREST is larger than EPS, QUAD invokes the PORT library’s centralized 
error handling facility,’ turning on the error state before returning 
control to the calling program. If the user has not taken prior action to 
recover from errors, an error message is printed and the run is termi- 
nated. The user who has taken prior action may test for the error state, 
and continue if desired. 

QUAD sets seven parameters to their default values and then calls 
RIQUAD. The default values should be adequate for almost all users, but 
the user wishing other values may call RIQUAD directly. RIQUAD also 
gives more information to the user about problems encountered during 
the integration. RIQUAD’s calling sequence is 
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CALL RIQUAD(F,A,B,EPS,HSAMPL,NFCALL,LYSTAK,KMAXEX, 
KDIVID,JPRINT,NUMINT,ANS,ERREST,KWARN) 


The default parameter values used by QUAD are 


HSAMPL 0.125 
NFCALL 2000 
LYSTAK 250 
KMAXEX 6 
KDIVID 4 
JPRINT 0 
NUMINT 3 


The same parameters are used for DQUAD, except that KMAXEX = 8. 

HSAMPL measures how finely f(x) must be sampled. No error estimate 
is believed unless the trapezoidal rule step size is HSAMPL |b — a| or 
smaller. Reducing HSAMPL improves the robustness of RIQUAD, but 
decreases its efficiency. Changing HSAMPL is useful for integrating os- 
cillatory functions, where there is some danger of aliasing. For {} cos 
(arx) dx, HSAMPL = 1/a is safe. The minimum number of sampling 
points of f(x) is roughly 2/HSAMPL. 

NFCALL is the upper limit on the number of sampling points of 
f(x). 

LYSTAK is the length of the stack for storing values of the function 
at the sampling points. Space for the stack is allocated within RIQUAD 
using a portable storage-allocation package.” 

KMAXExX is the maximum number of extrapolations allowed. Legal 
values are 4, 6, 8, 10, and 12. 

KDIVID is the minimum number of extrapolations required before 
dividing an interval. Legal values are 4, 6, 8,..., KMAXEX. 

JPRINT determines how much printing will be done. With JPRINT < 
1, there is none. With JPRINT = 1, the endpoints, attempted error tol- 
erance, answer, and error estimate are printed for each interval at- 
tempted. With JPRINT > 1, the T tables are also printed. 

NUMINT is the number of equal intervals into which [a,b] is divided 
to start. If NUMINT > 1, signularities can be recognized at x = a and at 
x = 6; if NUMINT = 1, asingularity can be recognized only at x = a. In- 
creasing NUMINT generally increases robustness while decreasing effi- 
ciency. 

KWARN is an integer warning flag, output from R1QUAD. It is zero if 
all went well. If KWARN is positive, it may have up to 6 digits, each with 
an independent meaning. Although ERREST may be greater than the 
requested EPS, it is still reliable. Each digit is zero unless a problem oc- 
curred; starting with the right-most digit, the problems are: 
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(t) The error estimate is limited by noise or roundoff, but is above 
the requested error. 

(it) The interval size is as small as is allowed. 

(it) As many intervals are stacked as is allowed. 

(tv) As many function values are stacked as is allowed. 

(v) As many function sampling points have been used as is al- 
lowed. 

(vi) The error estimate is larger than the requested error. 


4.1 Machine-dependent constants 


All machine dependency is isolated into four machine-dependent 
constants, which are set by RIQUAD. No reprogramming is necessary to 
run QUAD or RIQUAD on another computer. The constants are set using 
the portable machine constants program RIMACH of Ref. 7. 

DLARGE is used as error estimate before any error estimates are con- 
sidered believable. Its value is set to slightly less than the largest float- 
ing-point number, 0.1 RIMACH(2). 

DSMALL is used as the default magnitude of ro(x). Its value is slightly 
larger than the smallest positive floating-point number, 10 
RIMACH(1). 

DROUND Is used as the default magnitude of r;(x). Its value is set to 
50 times the largest relative rounding error, or 50 RIMACH(4). 

HSMALL is the smallest fraction of |b — a| used for a trapezoidal rule 
step size. Its value is the same as DROUND. 
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Recently we found that, among recursive digital filters using satu- 
ration arithmetic to contend with overflow, a fundamental difference 
exists between second and higher order filters: the latter may sustain 
large-amplitude overflow oscillations. In this paper we have derived 
a new criterion expressly designed for determining when a given 
high-order recursive system using saturation arithmetic is free of ov- 
erflow oscillations. The new criterion, which is easy to use, follows from 
this result: we associate with the given system two trigonometric 
polynomials in 6 of degree equal to the order of the given system; if any 
linear combination of the polynomials with nonnegative weights is 
positive for all 6 in [0,7] then the system is free of all nontrivial periodic 
oscillations. We prove that the new criterion subsumes certain well- 
known criteria, such as Tsypkin’s criterion, from the literature on 
nonlinear systems. To illustrate, three classes of special systems are 
investigated, and in each case the new criterion gives substantial im- 
provements. Finally, the new test is applied in the synthesis of high 
order sections for a realistic eighth-order system. 


1. INTRODUCTION 


Recently! we made the unexpected observation that, among recursive 
filters employing saturation arithmetic, a fundamental difference exists 
between second and higher order filters, namely, the latter may sustain 
large-amplitude overflow oscillations. This observation proved to be 
timely since it coincided with the awareness that economies of scale 
coupled with various recent developments make highly attractive the 
use of high-order sections in filter realizations. It has also come to light 
that the problem of possible large-scale oscillations is of interest not only 
in data filtering but in other areas where the natural structure is a 
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high-order recursive system, e.g., code converters (DPCM — PCM) and 
speech synthesizers. 

The economies of scale derive from the fact that the overflow detection 
and correction circuits, an expensive part of present-day filters, are as 
many as the number of sections employed; thus if a realization is com- 
posed of fourth-order sections rather than the conventional second-order 
sections, then the number of such circuits may be expected to be halved. 
The recent developments alluded to earlier refer to the almost simul- 
taneous developments of inexpensive, lower-power-consuming semi- 
conductor read-only memories, and the concept of distributed arithmetic 
blocks?? in which ROMs are used to implement digital filters. In a pi- 
oneering study R. B. Kieburtz‘ recently estimated that in a particular 
application a saving of about 30 percent in parts may be achieved over 
the conventional design through the use of fourth-order sections using 
saturation arithmetic and implemented by ROMs. 

Thus there is much to be gained if high-order sections can be used, 
and for this to happen it is first necessary to ensure that the highly de- 
structive overflow oscillations are not present in a particular design. It 
is apparent that there is a useful role for an effective criterion for de- 
lineating stable systems employing saturation arithmetic. It is possible 
to conceive of the situation where such a criterion is incorporated in the 
early design, i.e., the criterion is introduced as a constraint in the ap- 
proximation problem. The other possibility of an arithmetic different 
from saturation arithmetic to contend with overflow is not pursued in 
this paper. 

There do exist many such criteria in the literature on the stability of 
a class of nonlinear feedback systems (i.e., the systems in the Lurie 
problem‘) of which the one under consideration here is a member®-!!; 
the reader may consult Ref. 8 for a comparative evaluation of some of 
these criteria. These criteria are in some sense generalizations of Ny- 
quist’s criterion for linear feedback systems. The reader will find in Sec. 
5.2 a statement, in the context of the present problem, of Tsypkin’s 
criterion and the discrete circle criterion, two well-known examples of 
such criteria. Unfortunately it is known that when the nonlinearity in 
the system is the one associated with saturation arithmetic then these 
criteria, including Barkin’s criterion,’® are of limited utility since they 
are excessively pessimistic. Examples to this effect may be found here. 
Also telling is the fact that these criteria do not predict that all second- 
order systems using saturation arithmetic are free of oscillations,® a fact 
proven in Refs. 12—14 by arguments special to second-order systems. This 
is not totally unexpected in view of the fact that the systems-theoretical 
criteria apply to large classes of systems and nonlinearities and, con- 
comitantly, use relatively little information (restricted to the sector in- 
formation, symmetry, and monotonicity) about the nonlinearity. 
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Fig. 1—Schematic of unforced high-order filter employing saturation arithmetic. 


In this paper we have derived new criteria expressly designed for the 
system employing saturation arithmetic. Underlying the new criteria 
is the observation that certain unique passivity conditions are operative 
in the case of saturation arithmetic. Both the observations regarding the 
passivity conditions as well as the technique we use for deriving the 
criterion are believed to be new. The main ingredient in the derivation 
is the observation that the expressions associated with the passivity 
conditions in periodic solutions possess remarkable structure; namely, 
they are quadratic forms involving circulant matrices. 

The systems considered in this paper are of the form (see Figs. 1 and 
2) 


m 


Xn =F >: aj%n-j); n= Oa eee (1) 


ga 


F(a) 





Fig. 2—The saturation arithmetic nonlinearity. 
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where mm is the order of the system, {a;} are the coefficients, and F(-) is 
a nonlinear function associated with saturation arithmetic, namely 


F(o)=o if|o| <1 
sgn cif |o] = 1 (2) 


It is tacitly assumed that the underlying linear system in eq. (1) is ab- 
solutely stable, i.e., 


\n — z a;\"-i # Ofor all |A| > 1. (3) 
P 


Thus any nontrivial solution of eq. (1) will necessarily have either 1 or 
—1 as an element and consequently such solutions are referred to as 
overflow oscillations. Note that we are following convention in ignoring 
quantization effects in the description of the filter in eq. (1); in investi- 
gations of large-scale oscillations it is natural to focus on the gross non- 
linearity. 

The main result of this paper (Sec. 2.4) is simply stated: for a given 
system of order m with coefficients {a;}, we associate two polynomials 
of degree m in cos 0, namely, 


pi(8) = 1— cos é- % a;{cos j0 — cos (J —1)6} (4) 
j=l 
and 
p2(0) = 1+ cos d- 3 a;{cos j0 + cos § — 1)6}. (5) 
j=l 


If any linear combination of the polynomials with nonnegative weights 
is positive for all 6 in [0,7] then the system in eq. (1) with arbitrary initial 
conditions does not admit any nontrivial periodic solutions. Certain 
generalizations of this criterion are derived in Sec. VI. 

In Sec. 2.5 (“How To Use The New Test”) we show that the criterion 
may be used in a straightforward manner by one of two methods. The 
first method calls for plots of p1(0), p2(9) and p1(6)/p2(8) for @ in [0,7]. 
The second requires the consistency of a set of linear inequalities to be 
checked. The second method may also be used for the generalized cri- 
terion in Sec. VI. 

In Sec. III we examine three classes of special systems in detail. The 
results for the following canonical example! are typical: in a fourth-order 
system the poles are taken to be all real and repeated at p where |p| < 
1. For |p| = 0.669 overflow oscillations are proven to exist. Tsypkin’s 
criterion and the circle criterion guarantee the absence of oscillations 
for |p| < 0.384. The new test guarantees the absence of oscillations for 
|p| < 0.610. 
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In Sec. IV we apply the test to an eighth-order filter in the TDM-FDM 
translator, an extensively studied application of digital filtering. We find 
that this system can be lumped into two fourth-order sections employing 
saturation arithmetic, neither of which can sustain overflow oscillations. 
Both sections fail Tsypkin’s criterion and the circle criterion tests. 

Finally, in Sec. V we prove that the new criterion (i) easily gives the 
well-known result that overflow oscillations do not exist in second-order 
sections, and (ii) subsumes T'sypkin’s criterion. 


ll. THE CRITERION 
2.1 Passivity properties 


On account of the special form of the nonlinearity F the system in eq. 
(1) possesses certain simply stated but important properties which we 
interpret as passivity properties. The criterion we derive is a direct 
consequence of these properties. 

We may write eq. (1) as a linear system with a forcing sequence present 
by defining 


e, =F (x a;%n-i) — eo aj%n-j); n=0,1,... (6) 
1 1 
so that, from eq. (1), 
Xn = 3 Gitpap hen, n= 01,.%% (7) 
i=l 


The above nonhomogeneous linear recursion is used throughout the 
paper. Our procedure is to translate the important features of F(-) into 
tractable constraints on {ep}. 

Every solution of eq. (1) possesses the following properties. 


Proposition 1: 

Cn(Xn —Xn-1) $0, n =1,2,... (8) 
and 

Cn (Xn t+ Xn-1) <0, n=1,2,... (9) 


For proof we observe that if | 2 ajx,—-;| <1 then e, = 0 and both condi- 
tions are obviously valid. If 


3 


2D. ajXn-;>1 then x, =1 from (1) 


J 


Ut 
a 


and_ e, <0 from (6) 


and, of course, |x,—1| < 1. Thus, in this case eqs. (8) and (9) are valid. 
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Similarly, if Dajx,-; < —1 then x, = —1,e, > 0andas |x,-1| <1, the 
relations remain valid. 

Equation (8) may be viewed as stating that the forcing term e, has 
opposite sign from (x, — Xn—1) which we may interpret as the discrete 
analog of “velocity.” Thus locally the forcing term acts to reduce the 
velocity. Similarly interpreting (x, + x,—1)/2 as the local “distance,” 
eq. (9) states that locally the forcing term also acts to reduce the distance. 
For these reasons we view eqs. (8) and (9) as passivity properties. 

Note that the two above conditions imply the following weaker con- 
dition: 

gras 0, 1 = Lis (10) 


We find upon reflection that the latter condition is completely equivalent 
to the nonlinearity F in eq. (1) lying in the sector bounded by lines of 
slopes 0 and 1: 


0<F(o)/o <1 forall o. (11) 


This sector information is exploited in various criteria>-!! but the ad- 
ditional information in eqs. (8) and (9) is not. 


2.2 Equations for an oscillatory solution 


We state the equations associated with every oscillatory solution of 
period N of the nonhomogeneous recursion in eq. (7). We find that the 
equations in matrix form involve a circulant matrix. We put into per- 
spective some well-known results on circulants which are assembled in 
Appendix A. 

A periodic solution of eq (7) with period N has associated with it the 
following set of N equations involving the coefficients {a;}, the elements 


of the solution X1,Xo,...,Xn and the corresponding forcing terms 
Ey,Eo,...,EN: 
X= a,Xy + agXn-1 +++ + am XnN-m4+1 + Ey (12) 
Xq= aX, t+ agXn t-++ t+ amXn-m+2 + He 
Xy = a,Xn-1 t agXy—2 +-*++ + GmXN-m + En 
In matrix form, 
MX = E. (18) 


The interesting feature of the N X N matrix M is that it is a circulant!*!" 
since it is the following polynomial in the primitive N X WN circulant 
matrix P (see Appendix A for definition of P): 


M =I- > a;Pi (14) 
j=l 
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The circulant matrices have been extensively studied in the past and we 
are in the fortunate position of knowing a great deal of their eigens- 
tructure.' In particular, the eigenvalues of M are 


1— ¥ aje-wh2n/N, fk =1,9,...,N (15) 
j=l 

The eigenvectors of circulants are also known. The following remarkable 

property of the eigenvectors of circulants is of utmost importance in the 

paper (see next section): all N X N circulants have an identical set of 

eigenvectors, i.e., the eigenvectors do not depend on the constituents of 

the matrix. Thus, the eigenvectors of any N X N circulant are!: 


1 : : 
Uk = fe—ik(2a/N) @-i2k(27/N) | @—i(N-DR(2x/N), 1}* (16) 


k =1,2,...,N. Thus the real and imaginary components of the elements 
of each eigenvector are sequences of equiseparated samples of a sine 
function. Although these are complex vectors and circulants are not 
generally symmetric, the eigenvectors of circulants share an important 
property with eigenvectors of symmetric matrices in that they form an 
orthonormal set, i.e., 


u,Uy = dz. (17) 
In matrix notation, 
UU =I (18) 


where the eigenvectors {u;} have been arranged as columns of the matrix 
U. 


2.3 Another representation of the passivity properties 


We combine the above information with the passivity properties stated 
in Proposition 1 to obtain a compact and useful representation of the 
passivity properties that are valid if an oscillatory solution to (1) exists. 
As in the preceding section an oscillatory solution is assumed to be of 
period N. 

Note that we may write 


(Xy,X1,...,Xn-1) = (Xi Xo,...,XN)P! = XP’ (19) 


t Recently we have had another occasion!8 to use the eigenstructure of the matrix M. 
Willson!® investigates the matrix M from a different angle. 

! We denote the conjugate transpose by the superscript *. In the case of real matrices it 
is also denoted by the superscript ’. 
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where P is the primitive N X N circulant. Thus 
N 


En(Xn — Xn-1) = X’(I— P’)E 
n=1 
= X’/(I — P’)MX, from (18) 
=X’ E ~ 3 a;Pi- PN-14 5° P| Xx (20) 
j=l j=l 


where in the final step we have used, in addition to the expression for 
M, the relations P’ = PN—! and PN =I. The key observation about eq. 
(20) is that the matrix there, being a polynomial in P, is a circulant. 

We undertake a convenient change of coordinates to diagonalize the 
matrix in eq. (20). Let 


Z = U*X (21) 


where U is, as in Sec. 2.2, the unitary matrix of eigenvectors of N X N 
circulants. Denoting the known eigenvalues (see Appendix A) of the 
matrix in eq. (20) by uy, k = 1,... ,N, we obtain 


N N 
zy Ei xy = Xn-1) = > |Z: | 2 Re up (22) 
Now, 
Re up, = 1 —cos {k(2n/N)} — 3° a;[cos {jk2n/N} — cos {(j — 1)k2x/N}] 
j=l 


(23) 


To put eqs. (22) and (23) into the most convenient form, define the 
polynomial p;(-) where 


pi(@) =1—cosd- s a;{cos 8 — cos G — 1)6} (24) 
j=l 
We then have 
N N 
» E,(Xn = Xn—1) = p> |Zp | 2p1(k2x/N). (25) 


We proceed in identical fashion to obtain a similar expression corre- 
sponding to the other passivity condition in Proposition 1. Note that 


N m 
Y En(Xn + Xn-i) =X’ 1- ¥ ajPi+ PN 
j=l 


n=1 


= ajPi-1| X (26) 
i=1 


j 
Because of the previously mentioned (and crucial) property that all N 
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X N circulants have identical sets of eigenvectors, the diagonalizing 
transformation is same as the one undertaken previously in eq. (21). 
Hence 


N N 
> E,(Xn °F Xn-1) = a |Z, | 2Re AR (27) 
n=1 k=1 


where we have denoted the eigenvalues of the matrix in eq. (26) by {Ax}. 
Here 


Re Ay = 1 + cos {k2a/N} — a a;[cos {jk2a/N} + cos {7 — 1)k2x/N}] 
j=l 


28 
k =1,2,..., N. Thus for the final form we obtain a 
ZX EnXn+ Xn-a) = ¥ [Zel2pa(h2a/N) (29) 

where the polynomial po(-) is defined to be 
p28) = 1+ cos 8 = a;{cos j6 + cos G — 1)6} (30) 


Now certainly Proposition 1 implies that 
N N 
Dv En(Xn — Xn-1) <0 and Yo EA(Xn+Xn-1) <0 (31) 
n=1 n=1 


The above, together with eqs. (25) and (27), yields: 


Proposition 2: If a periodic solution of period N with elements 
(X1,Xo,...,Xy) exists for the recursion in eq. (1) then 


Py |Z, |2p1(k27/N) <0 (32) 
and 
Ds |Z;,|?2po(k2x/N) <0 (33) 


where Z, as given in eq. (21), is a transform of X and p;(6) and p2(9), given 
in eqs. (24) and (30), are two polynomials in cos 0 of degree equal to the 
order of the system of eq. (1). 


For a fourth-order system (m = 4) the two polynomials are 
pPi(@) = (1 + ay) — (1 + a1 — ag) cos 6 — (ag — a3) cos 20 
— (a3 — a4) cos 86 —a4cos 46 (34) 
and 
p26) = (1 — ai) + (1 — ay — ag) cos 6 — (ag + a3) cos 20 
| — (ag + a4) cos 80 —a4cos 40 (35) 
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The polynomials for second- and third-order systems are obtained from 
the above by setting a3 = a4 = 0 and a, = 0, respectively. 

In Fig. 3a and b we have plotted p;(@) and po() for a particular 
fourth-order system. 
2.4 The main result 

It is only a short step from Proposition 2 to the main result which 
is 
Theorem 1: If for any a; > 0 and ao = 0, 

a1p1(9) + agpo(9) > 0 for all 6 in [0,7] (36) 


then nontrivial periodic oscillations do not exist as solutions to eq. (1). 


Proof: The proof is by contradiction. Suppose a nontrivial (i.e., X # 0) 
periodic solution of period N exists and also that the hypothesis of the 
theorem is valid. Then for such a solution 


N N 
ay > |Z;,|2pi(R27/N) + ae = |Z, | 2po(k2x/N) 


N 
= Pa |Z; | 7{a1p1(k27/N) zm agpo(k2x/N)} 


>0 (37) 
from the hypothesis. However, from the passivity conditions summarized 
in Proposition 2, 


N N 
ay X [Zn |2p(k2x/N) + a2 ¥ (Zn |2palk2n/N) <0 (88) 
=] k=1 


which is a contradiction. QED. 


Note that if it is desirable to know only that oscillations of a particular 
period N do not exist for eq. (1) then the following is a sufficient condi- 
tion: 

There exist 
a, 20, a> Osuch that aypi(k27/N) + agpo(k2ar/N) >0 (89) 
fork =1,2,...,N. 


2.5 How to use the new test 


Given an mth-order system, there are two simple and straightforward 
ways in which the above result may be used to determine whether the 
system does not admit overflow oscillations. 

The first method requires p1(0), p2(6), and p;(0)/p2(8) to be plotted 
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a 3—Plots of polynomials pi(@), p2(9), and a 1p,(8) + aspe(8) for the following 
fourth-order system: a; = 1.1015710, ag = —1.6571120, a3 = 0.7733805, a4 = —0.45135546. 
In (c), ay = .92348761, ag = .16965636 


for 6 in the interval [0,7]. When such plots are available the first step is 
to see if there is any 0 for which both p;(@) and po(8) are negative; if this 
is the case the hypothesis of Theorem 1 obviously cannot be satisfied and 
the test is automatically failed. Assuming that this is not the case we find 
upon reflection that the hypothesis of Theorem 1 is satisfied if and only 
if 

[p1(9)/p2(8)] < min ) [P1(9)/p2(9)] (40) 


xX 
{6|p1(9)>0,p2(8) <0} {0|p1(9)<0,p2(0)>0 


In fact, if the above is true the interval defined by the left- and right-hand 
sides of (40) is not empty and the hypothesis of Theorem 1 is satisfied 
by taking a; = 1 and —a¢ to be any value in the interval. 

In summary the procedure is as follows: first check to see if p; and po 
are both negative at the same point. If so, then the test is failed; if not, 
proceed to determine the intervals where {p;(@) > 0,p2(6) < 0} and where 
{p1(8) < 0,p2(8) > 0}. The test is passed (i.e., no overflow oscillations exist) 
if and only if the maximum of p(9)/po(8) in the first interval is less than 
the minimum of p;(0)/po(@) in the latter interval. 

In the second method we finely discretize the interval for 6, [0,7], and 
evaluate p;(0) and po(8) at all the discrete points {0;}. Testing for the 
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hypothesis of Theorem 1 then amounts to testing for the consistency of 
the following set of linear inequalities 


e as) | 5 O pi(81) py(99) - - - pi(z) 
Pea ety p21) pele) - - - po(m) 


(There is no loss of generality in assuming that the right-hand side is as 
specified above.) There are standard procedures2%?! for testing for 
consistency of linear inequalities. In any case, Phase 1 of any commer- 
cially available lear programming package does precisely this. If a linear 
programming package is used then the following (dummy) functional 
may be used in the program: minimize (a1 + ag). 

The above method is easily adapted to the generalization of the cri- 
terion which is developed in Sec. VI. 


|Pa1.v (41) 


lil. EXAMPLES 


We consider three classes of examples in some detail. In each case we 
tested the criterion by following the second method outlined above. We 
used a linear programming package (written in machine language) made 
available to us by A. M. Odlyzko; the interval [0,7] was subdivided into 
100 intervals. In every case the computation time was of the order of a 
second. 


3.1 Example 1: third-order system with repeated real roots 


In this class of examples we take the coefficients to depend on a real 
number p, |p| < 1, in the following manner: 


a,=—3p, a2=—8p?, a3 = —p* (42) 


A third-order system with the above coefficients corresponds to an 
underlying linear system with characteristic polynomial (A + p)?°, i.e., 
the linear system possesses three real roots all repeated at —p. In the 
investigation reported in Ref. 1 we found this class of systems to be in- 
teresting for various reasons. Also, for |p| close to 1 the behavior of the 
system is to some extent representative, at least with respect to oscilla- 
tory behavior, of low-pass systems and high-pass systems, depending 
upon whether p is negative or positive respectively. 
In Ref. 1 we showed for system (1) that 


|p| = 0.858 = period-3 oscillations exist (43) 
Tsypkin’s criterion and the circle criterion (see Sec. 5.2) give 
|p| < 0.500 = no overflow oscillations exist (44) 
An application of the new test yields 


|p| < 0.785 = no overflow oscillations exist 
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Thus, in this class of examples the new test makes a substantial contri- 
bution in reducing the indeterminate region to 0.785 < |p| < 0.858. 


3.2 Example 2: fourth-order system with repeated real roots 


The class of examples considered here is a natural extension to a higher 
order, m = 4, of the class considered in the previous example. Again all 
the coefficients are determined by one real parameter p where |p| < 
1: 


a, = —Ap, a2= —6p?, a3 >= —4p3, a4 = —p4 (45) 
Thus in this example the underlying linear system possesses four real 
roots, all repeated at —p. 

By examining the natural set of four equations associated with a pe- 
riodic solution of period 4, see eq. (12), it is easy to see that a periodic 
solution with elements (1,1,—1,—1) exists if and only if 

(a4 — a9) > 1+ |a; — a3] (46) 
Thus, we find on substituting for the a’s that 
|p| = 0.669 = period-4 oscillations exist (47) 
Tsypkin’s criterion and the circle criterion give 
|p| < 0.384 = no overflow oscillations exist (48) 
Application of the new criterion gives 


|p| < 0.610 = no overflow oscillations exist (49) 


Thus we find that in this example too the new criterion makes an ef- 
fective contribution in determining the region of stability. 


3.3 Example 3: fourth-order filter for sample rate conversion 


The example we consider now, a fourth-order system, was designed 
originally for interpolation and filtering for a terminator in a local digital 
switch.4 We have reported previously! that in its original form the filter 
using saturation arithmetic sustained overflow oscillations. Here we vary 
one of the parameters in the design in order to estimate the modification 
required to guarantee the absence of oscillations. We find that the req- 
uisite variation is large. However, in the process we obtain a measure of 
the effectiveness of the new criterion. 

The example we consider has two pairs of complex poles 


Azo = pie*!81, Aga = poe #2 (50) 


(The coefficients of the system are not of much interest; however, they 
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may be obtained from the information given below.) Also 


p1 = 0.786427817, 6, = 37.309784226 degrees (51) 
and 


J. = 39.675296075 degrees 


We vary pe keeping pj, 91, and 6 fixed at the above values; in the original 
design po = 0.952851183. 

In (46) we have given a condition for the existence of limit cycles of 
period 4 with elements (1,1,—1,—1). Translating (46) to the present ex- 
amples gives 


p2 => 0.671 = period-4 oscillations exist (52) 
Tsypkin’s criterion and the circle criterion give 
po < 0.070 = no oscillations exist (53) 


An application of the new criterion gives 
p2 < 0.665 = no oscillations exist (54) 


This is a rather striking example of the effectiveness of the new criteri- 
on. 


IV. AN APPLICATION 


Here we examine a particular eighth-order system* which has been 
used in an applied research project22 on a TDM/FDM translator.?? The 
latter, a system for translating between analog frequency-division and 
digital time-division signals, is an extensively studied application of 
digital filtering. The eighth-order system has been designed to function 
as a low-pass filter with a sampling frequency of 8 kHz and a cutoff fre- 
quency of 2 kHz. Our object here is to demonstrate through an applica- 
tion of the new criterion that it is possible to design the filter as a cascade 
of two fourth-order sections both employing saturation arithmetic such 
that no overflow oscillations are sustained in either section. At least as 
far as overflow oscillations are concerned the margin of safety is adequate 
so that small changes in the coefficients due to quantization of coeffi- 
cients, for example, are not going to cause overflow oscillations to appear. 
It should be emphasized that the result here is not a substitute for a 
design study and the structure suggested may well turn out to be unac- 
ceptable on grounds not related to overflow oscillations. 

The system has four pairs of complex poles; the modulus (p;) and 


* IT am grateful to V. B. Lawrence for bringing this system to my attention. 
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argument (+0;) of each pair is as follows: 
p1 = 0.5115846 , 6, = 32.870 degrees 
p2 = 0.980274196, 42 = 80.828 degrees 
p3 = 0.75259969-, 63 = 64.482 degrees 
pa = 0.892679, 64 = 75.297 degrees 


We group the first and second pairs of poles together to form one 
fourth-order section and the remaining pairs to form the second 
fourth-order section. The resulting coefficients of the two sections are, 
respectively, 


a; =1.1718731, ag = —1.4912153, ag = 0.9075846, 
a4 = —0.2514954 (56) 


a; =1.1015710, ag = —1.6571120, ag = 0.773838085, 
a4 = —0.45135546 (57) 


Both sections pass the new test. For the first section it may be ascer- 
tained that with 


a1 = 6.0819413 and ag = 0.07538601 (58) 


the hypothesis of Theorem 1 is satisfied. In fact, the polynomial p,(6) 
is itself positive everywhere except at 0 = 0, where its value is 0. However, 
p2(0) > 0. Thus, any positive choice of a; and ag chosen suitably small 
will satisfy the hypothesis of Theorem 1. 

For the second section (57), a choice of a; and a2 for which a 1p1(6) + 
aope(0) > 0 for all 6 is 


ay = .92348761 and ag = .16965636 (59) 


Plots of p1(0), p2(9), and a1p1(9) + a2po(@) for the second section are 
displayed in Fig. 3. 

It is noteworthy that both sections fail Tsypkin’s criterion and the 
circle criterion. 


V. SOME IMPLICATIONS OF THE MAIN RESULT (THEOREM 1) 
5.1 Overflow oscillations do not exist in second-order systems 


It is well known!?2:!3.14 that when the order of the system in eq. (1) is 
two, then overflow oscillations are not sustained. The proofs of this result 
are rather special to second-order systems and to the saturation arith- 
metic. On the other hand, there are the frequency-domain criteria®-!! 
for stability which are systems-theoretical results applicable to large 
classes of nonlinearities and systems of arbitrary order. However, we may 
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infer from the results in Ref. 8 that these criteria do not give the result 
that all second-order systems are free from overflow oscillations. 

We show that the criterion in Theorem 1 does give the well-known 
result on second-order systems. Our result is given in Proposition 3. [It 
is assumed that |a2| < 1 and 1 — |a;| — ag > 0; these relations are 
equivalent to eq. (3), i.e., the underlying linear system is stable.] 


Proposition 3: Let m = 2 in eq. (1). Also let 
ay = (1 +a; — ae) > Oand ao = (1 — ay — ao) > 0 (60) 
Then, 
a1P1(0) + aepe(0) > 0 for all 6 (61) 


The proof of this result is in Appendix B. The above in conjunction 
with Theorem 1 shows that oscillations are not sustained in second-order 
systems. 


5.2 Tsypkin’s criterion and discrete circle criterion are subsumed by new 
criterion 


The object here is to show that the new criterion subsumes both 
Tsypkin’s criterion® and the discrete circle criterion!! when the latter 
criteria are used to determine the nonexistence of oscillations in eq. (1). 
The two closely related frequency-domain criteria are identical when 
applied to the system in eq. (1). 

Tsypkin’s criterion® is as follows in applications to systems like eq. 
(1) where the nonlinearity F satisfies 


Kymin < F(o)/o < Kmax forall o (62) 
If 
(i) ° ajz~4/[1 — Kmin © ajz7/] is finite for all |z| > 1 (63) 
(iz) Kr... -k..7 Re [> aje—#/9/(1 — Kinin © aje~J®)] > 0 for all 


6 in [0,27] (64) 


then lim x, = 0; in particular, oscillations do not exist. 


In the case of eq. (1) where F is the saturation nonlinearity, 
Kmin=0 and Kyax=1 (65) 


so that the effective restriction is (64) which reduces to 


1-  Saj;cosj@>0 forall (66) 
J=1 
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When (65) holds the discrete time version of the circle criterion! is 
identical to the above condition. 

In Theorem 1 let a; = ag = 1. Then from the defining relations for 
pP1(8) and po(@) in (24) and (30) respectively, we find that 


ap 1(6@) + aepo(6) =l1- x aj; COS j0 (67) 
j= 


Thus, as previously asserted, if either Tsypkin’s criterion or the circle 
criterion is satisfied, i.e., (66) is valid, then the hypothesis of Theorem 
1 is also satisfied. 


Vi. A GENERALIZATION OF THE MAIN RESULT 


The reader will recall that the main result, Theorem 1, is a direct 
consequence of the rather special passivity properties, stated in Sec. 2.1, 
which are implied by the special features of the saturation nonlinearity 
F. Another key ingredient is that the passivity conditions imply ine- 
qualities on quadratic forms involving circulants. We show here that 
many conditions akin to the ones in Proposition 1 are valid by virtue of 
the properties of the saturation nonlinearity. All or some of these may 
be used to augment the passivity conditions used so far so as to obtain 
improved criteria for the nonexistence of oscillations. 

The following generalized passivity conditions exist* for any / = 1: 


Qn(Xn —Xn-1) <0 n=lit+ti,... (68) 
Cn (Xn + Xn-1) <0 n=l, +1,... (69) 


where {x,,} is any solution of eq. (1) and {e,,} is obtained from the solution 
through eq. (6). The proof is similar to that of Proposition 1. Thus in 
Proposition 1 we have used only a very small subset (/ = 1) of all the 
above conditions. 

The interesting fact is that each of the expressions in the above con- 
ditions summed over N, where N is the period of any periodic solution 
of (1), is equivalent to a quadratic form involving a circulant. Thus if X 
= (X1,Xo9,...,Xn)’ are the elements of the periodic solution and 
(£y,E9,...,En) are the corresponding forcing terms, see eqs. (12) and 
(13), then 


N hs > 
Y Ey(Xn ~ Xn) =X | 1 F ajPi- P-!— FP] X 
ol eH j=l 


(70) 


* The generalized passivity conditions are also valid for negative values of | although we 
do not make any explicit use of this fact. 
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N m m 
De En(Xn + Xn-1) = X’ E —-YaPi+P$+> ajP-'+i| X 
n=1 j=l 73 


(71) 
for] =1,2,.... Hence by transforming X to Z where Z = U*X, U being 


the unitary matrix of eigenvectors of N X N circulants, we obtain for | 
= 1,2,... 


N N 
u BE MXq— Xn) = 2 |Z, | 2pi(k27/N) (72) 
and 
N N 
> Ey (Xn + Xn-1) = 2 |Z |2p’1(k27/N) (73) 
where 
pi(0) = 1 — cos 10 — 3° a;fcos j0 + cos (j — 1)6} (74) 
a1 
and 
p(0) = 1+ cos lé— = a;{cos j0 — cos (j — 1)6} (75) 
jm 


Thus p;(@) and p(6) defined in Sec. 2.3 correspond to p;(6) and p;(6) 
respectively in the present notation. 

Certainly the generalized passivity condition in (68) and (69) imply 
that the expressions in (72) and (73) are nonpositive. We thus arrive at 
the following generalization of Theorem 1: 


Theorem 2: If any convex linear combination of 
P1(9),p3(8),P2(9),po(6), ... is positive for all 6 in [0,7], then the system 
in eq. (1) does not have any nontrivial periodic solutions. 


In experiments involving fourth-order systems of practical interest 
we have not found the use of the above generalized criterion to make any 
material difference in delineating the stable systems. In these investi- 
gations we used a linear programming package (Sec. 2.5) to apply the 
test in Theorem 2 with up to six polynomials (the leading six polynomials 
of Theorem 2) being used. However, it is quite possible for substantial 
improvements to exist in other cases. 
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APPENDIX A 
Circulant matrices 


For completeness we collect here some of the well-known properties 
of circulants which are used in the paper. The interested reader may refer 
to Muir!® and Grenander and Szego!’ for further details and applica- 
tions; Ref. 15 concisely lists some of the main properties. 

We let P denote the primitive N X N circulant: 


0 < = — = 0 1 
1 0 — _— = — 0 
P= 0 1 0 a 0 (76) 
0 —_ ae — a 1 0 
Note that 
PN=] (77) 
and that 
p’=PA-l!=Pp-! (78) 


A polynomial of arbitrary degree in P is a circulant. An N X N circulant 
C, 


N-1 
C= >> c;P) (79) 
j=0 
has as its eigenvalues 
N-1 7 
cje~k2n/N k= 1,2,...,N (80) 
j=0 
All N X N circulants have as eigenvectors uz, k = 1,...,N, given ineq. 
(16). The matrix U with the eigenvectors as columns is unitary, i.e., 
UU=T1 (81) 
APPENDIX B 
Proof of proposition 3 


We prove here the assertion in Proposition 3, namely, for second-order 
systems 


q(6) & (1 5 ay, a2)p1(8) + (1 — a, ao) po(8) >0O forall 6 (82) 
For second-order systems 


p1(0) = (1 +. a1) — (1 + ay — a2) cos 6 — ae cos 20 (83) 
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and, 
p2(6) = (1 — a) + (1 — ay — ae) cos 6 — ae cos 20 (84) 
We find upon substitution that 
q(0) = —4a9(1 — a2) cos? 6 — 4a;(1 — ag) cos 9 + 2(1 + a? — a8) (85) 
First observe that at the corner points q is positive: 
q(0) = 2(1 — ay — ag)? > O and q(x) = 2(1 +a, —a2)2>0 (86) 


Through differentiation we find that minima of qg(@) occur in the interior 
of the region [0,7] if and only if 


|ay| < —2a9 (87) 

and that at a minimum 8, 
cos 0 = —a,/2a (88) 

Evaluating q at such a point we obtain 
p —(l+a 

g(@) =P) pat +a) + daha] (89) 

2 

>0 
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Part 1—Narrowband-to-Wideband Link 
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The performance of a tandem connection of narrowband and wide- 
band speech communication systems is evaluated. Specifically, the 
narrowband system consists of a conventional Linear Predictive Coding 
(LPC) vocoder operating at a bit-rate of 2.4 kb/s and the wideband 
system consists of a Continuously Variable Slope Delta modulator 
CVSD operating at a bit rate of 16 kb/s. In Part 1 of this paper the 
properties of the narrowband-to-wideband link are investigated and 
in Part 2 the properties of the wideband-to-narrowband link are in- 
vestigated. In part 1 the SNR (signal-to-quantizing noise ratio) of the 
CVSD coder ts analyzed over a 50-dB variation of the input signal levels 
and for a variety of source excitations for the LPC synthesizer. It is 
shown that SNR improvements in the CVSD coder of 2 to 2.5 dB are 
possible in the slope overload region of the coder by modifying the 
source excitation of the LPC synthesizer and by preprocessing the input 
signal to the coder with an allpass filter. Both methods aid in reducing 
the peak factor (peak-to-RMS level) of the input speech to the coder. 
Subjectively, however, only slight improvements in quality, if any, were 
observed with these modifications. 


Il. INTRODUCTION 


Agencies of the United States government are currently formulating 
plans for an extensive digital secure voice communication network. In 
this network, a substantial fraction of the signals will be transmitted over 
“wideband” circuits at 16 kb/s. Owing to severe bandwidth constraints 
in some parts of the network, however, there will also be “narrowband” 
speech links in which the transmission rate is 2.4 kb/s. In preliminary 
plans, the wideband code format is CVSD (Continuously Variable Slope 
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Fig. 1—Narrowband-to-wideband link. 


Delta modulation) and the narrowband code format is LPC (Linear 
Predictive Coding). 

Both of these coding methods have been studied extensively, and their 
performance over single transmission links (involving one encoding 
operation and one decoding operation) is now well understood.!:? In 
addition to creating single links, however, the proposed communication 
network will establish tandem connections containing both narrowband 
and wideband links. It is not clear a priori that two systems, each de- 
signed for single-link operation, will interact in tandem to provide ac- 
ceptable overall quality. Existing knowledge of LPC and CVSD is of 
limited value in predicting tandem performance and yet the viability 
of the proposed network depends on adequate performance of tandem 
as well as single circuits. It is the purpose of this paper to describe the 
properties of CVSD and LPC that influence the performance of the nar- 
rowband-to-wideband connection shown in Fig. 1. A companion paper 
deals with the complementary wideband-to-narrowband connection. 

Our study focuses on issues that arise in tandem links but not in in- 
dividual circuits. In particular, in this paper we investigate the effects 
of the narrowband channel on CVSD signal-to-noise ratio (SNR). In doing 
so we have measured the SNR of the CVSD coder with an original speech 
input and compared it with the SNR when the CVSD input is LPC syn- 
thesized speech with a conventional (impulse) excitation source (during 
voiced intervals). With a view to improving the quality of tandem cir- 
cuits, we have also investigated the effect of allpass filtering the LPC 
output and of using broadened excitation sources for voiced sounds in 
the LPC synthesizer. 

The studies have been carried out by means of computer simulations 
on a Honeywell DDP 516 computer. In the narrowband-to-wideband 
tandem we have measured SNR as a function of CVSD input level for a 
variety of interface and LPC synthesizer source configurations. For each 
condition (i.e., a given input level, synthesizer source and interface) we 
have recorded two sentences transmitted through the tandem link. The 
SNR measurements as well as informal listening experience suggest that 
CVSD is a critical element in this tandem connection. It has been shown 
that combinations of interface filter and modified synthesizer source 
are effective (to some extent) in improving overall quality when the CVSD 
input level is high. In this case the delta modulator is subject to sub- 
stantial slope overload. The overload is reduced both by prefiltering and 
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Fig. 2—Block diagram of LPC-to-CVSD link. 


by broadening of the LPC synthesizer source because both of these 
methods reduce the ratio of peak-to-rms level of the signal at the CVSD 
input. Of the two methods, adjustment of the LPC excitation is more 
effective but does require some modification of the LPC link. All-pass 
filtering the LPC output signal has the advantage of being external to 
both CVSD and LPC. 


ll. OVERVIEW OF THE NARROWBAND TO WIDEBAND LINK 


In this section we discuss the elements of the narrowband-to-wideband 
tandem connection. We will first review the basic operation of the LPC 
vocoder and the CVSD coder and will then discuss issues involved in 
connecting these two systems in a tandem link. After establishing a basic 
understanding of the various elements in this link, we will discuss factors 
which affect the performance of this connection and ways in which this 
performance can be improved. 

Figure 2 shows a more detailed block diagram of the overall tandem 
connection. The narrowband system consists of an LPC analyzer and a 
pitch and voiced/unvoiced detector. The parameters from these two 
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analyses are used by the LPC synthesizer to resynthesize the speech 
waveform. An allpass network may be used for further post processing 
of this waveform. The details of this network will be discussed in a later 
section. 

As shown in Fig. 2, the wideband system consists of a bandpass filter 
to prevent aliasing, the CVSD coder and another bandpass filter that 
suppresses CVSD quantizing noise. Gains G and 1/G are used in mea- 
suring the dynamic range (i.e., variations in performance as a function 
of signal level) of the CVSD coder. 

The basic sampling rate of the narrowband system is 10 kHz and the 
sampling rate of the wideband system is 16 kHz. In order to interface 
these two systems a sampling rate converter is used. The details of this 
conversion will also be discussed in this section as well as the conversion 
from 16 kHz to 10 kHz which is required in the wideband-to-narrowband 
tandem connection. 


2.1 The wideband system (CvspD) 


Figure 8a is a block diagram of the CVSD coding process. The input 
speech signal is called x(t). An approximation signal y(t) is generated 
in the encoder feedback loop and at the kth sampling instant (t = RT, 
T = 1/16000 sec), the transmitted signal is b; = 1 if 


x(kT) = xp > ye = y(RT) (1) 


Otherwise b; = —1. A positive output causes y(t) to increase during the 
next sampling interval making y;,+1 attain the value 


Yr+i = ay, + H(1 — a)Ay (2a) 


where a is the leakage of the approximation signal integrator and A; = 
A(kT) is the kth step size. A negative output, b, = —1, results in 


Yeti = ay, — H(1 — a)A, (2b) 


The step size is obtained from another integrator which processes the 
output of an overload detector. The overload detector has output V when 
the three previous CVSD outputs are identical (all 1 or all —1). Otherwise 
the output of the overload detector is 0. To ensure that the minimum 
step size is nonzero a small quantity V, is added to the output of the 
overload detector. Thus, the step size satisfies the relation 


Arti = BA, + (1 — B)(V + V4) (3) 


when three previous outputs are identical where @ is the leakage of the 
step size integrator. Otherwise, 


Arti = BA, + (1 — B)V1 (4) 
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Fig. 3—(a) Block diagram of CVSD coder and (b) circuit implementation of CVSD 
coder. 


Figure 3b shows the circuit implementation of these operations. In 
the digital logic, point A is the output of the overload detector. It is an 
open circuit when the three previous bits are all 1 or all —1. This open 
circuit condition allows the 1 uF capacitor to charge toward + V through 
R1 and R3. When the last 3 bits are not identical, point A is grounded 
and the capacitor discharges to ground through R1 and R4. The poten- 
tiometer R6 establishes V; the minimum voltage on Cl. 

When b; = 1, point C is grounded and point D is an open circuit. The 
gain of amplifier A2 is H = 3 and the voltage at point E is 3A. When b; 
= —], Cis open circuited and D is grounded causing the voltage at E to 
be —3A. Thus the integrator R2-C2 charges toward +3 times the voltage 
on capacitor C1, depending on whether b; = +1. 

The time constant of the step size integrator is 5.69 ms (1 uF X R1 + 
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R3 or R4 in parallel with R5 and with the input impedance of A2 which 
is 20 kQ). The step size coefficient is therefore 

= 
16,000 


The time constant of the approximation signal integrator is 1 ms which 
gives 


B = exp (- / 5.69 X 10-8) = .99 (5) 


1 
= exp(—-—— 10-*) = 94 6 
ss exp( an) (6) 


In the computer simulation, speech is represented as a 16-bit integer 
between —32,768 and 32,767, so that a value of V = 32,767 is equivalent 
in hardware to a peak speech input equal to the supply voltage. In our 
studies we have provided for a wide dynamic range of step sizes, V/V 
= 200 so that V,; = 164. 

Thus eggs. (3) and (4) are, numerically, 


Ari = 99 A, + 329 (7) 
when three outputs are identical and 
Ar+1 = -99A; + 1.64 (8) 
otherwise. Similarly eqs. (2a) and (2b) are, 
Yr+i = 4y, + .18A, (9) 
if xz > y, and 
= .94y, — .18A;, (10) 


otherwise. 


2.2 The narrowband system (LPC) 


The narrowband system consists of a Linear Predictive Coding (LPC) 
system based on an all-pole model of the speech production mechanism. 
The all-pole model implies that within a frame of speech, the output 
speech sequence is given by 


Sn = > ApSn—p + Gun (11) 
k=1 


where p is the number of poles, u,, is the appropriate input, G is the gain, 
and the a;’s are the LPC coefficients that represent the spectral char- 
acteristics of the speech frame. For a voiced speech segment, u,, is a Se- 
quence of pulses separated by the pitch period. If the segment is un- 
voiced, pseudorandom noise is used as input. 

In our study, the LPC coefficients were calculated by the autocorre- 
lation method with p = 12 (Ref. 2). The analysis was performed every 
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20 msec (50 times/sec) across overlapping 300 sample (30 msec) Ham- 
ming windowed speech frames. The pitch detection and V/U (voiced/ 
unvoiced) decision is based on the modified autocorrelation method.? 
The effects of pitch and V/U analysis do not in general influence the 
performance of the narrowband-to-wideband link. In the reverse link 
(wideband to narrowband), however, the pitch and V/U analysis is 
strongly affected by the performance of the wideband system. Therefore 
we will discuss the pitch and V/U analysis in the accompanying paper 
on the wideband-to-narrowband link. 

Since the stability and characterization of the LPC synthesizer is ex- 
tremely sensitive to small perturbations in the LPC coefficients, it is not 
possible to achieve low-bit-rate coding by transmitting the LPC coeffi- 
cients.2 However, by transmitting either the log area coefficients or the 
parcor coefficients, a 2.4-kb/s vocoder is readily achieved. The log area 
coefficients are related to the LPC coefficients by 

1+k; 

8 = logs h; (12) 
where the k;’s are termed the parcor coefficients.2 If we denote a; as 
the ith linear prediction coefficient for a jth-order linear-prediction 
model then 


k; =a; (i) (13) 
The parcor coefficients have the very important property that if 
|k;| <1 YS] eg D (14) 


then it is guaranteed that the linear prediction synthesizer is stable.” 
Thus, small perturbations in the parcor coefficients or log area coeffi- 
cients will not affect the stability of the synthesizer, and morever these 
small perturbations will not seriously alter the spectral characterization 
of the speech segment.® Since the log area coefficients are slightly less 
sensitive to quantization error® they were transmitted in the narrowband 
system. 

The quantization of the LPC control signals (pitch, gain, and the g;’s) 
was accomplished by ADPCM (Adaptive Differential PCM) techniques.® 
In this scheme, the value of a particular control parameter in the nth 
frame is initially estimated as equal to the transmitted values of the 
parameter in the (n — 1)st frame. The difference between this predicted 
value and the actual parameter value is then quantized using a gamma 
or laplace quantizer with an adaptive step size. Complete details of 
the adaptation scheme and the quantization method are given in ref. 
4, 

The bit allotment in the narrowband link is as follows. The pitch and 
gain information is encoded with 3 bits/sample each. The first six log area 
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ratios £1, £9,..., 8g are each encoded with 4 bits/sample and g7, gg,..., 
£12 are encoded with 2 bits/sample. One bit/frame is used to transmit 
the V/U decision. This gives a total of 43 bits/frame or a transmission 
rate of 2.15 kb/s. Another 5 bits/frame are used for transmission of ini- 
tialization information and frame synchronizing information giving a 
total of 48 bits/frame or a total transmission rate of 2.4 kb/s for the 
narrowband system. 


2.3 Bandpass filters 


The bandpass filters in Fig. 2 are all identical and are used to limit the 
bandwidth of the signal to the range 200 Hz to 3200 Hz. The third 
bandpass filter below the block diagram of the wideband system is used 
for compensating the group delay of the input signal of the CVSD in order 
to make meaningful SNR measurements on the CVSD. 

The bandpass filters are eighth-order recursive elliptic filters with a 
passband ripple of 0.25 dB and a stopband attenuation greater than 35 
dB. The average group delay of the filters is 0.825 msec in the passband 
and it peaks to 7 msec in the lower transition band. Figure 4 shows the 
log magnitude response (dB), group delay, and impulse response of these 
filters. 


2.4 Sampling rate conversion 


In the tandem connections it is necessary to convert the sampling rate 
of the signal from 10 kHz to 16 kHz and from 16 kHz to 10 kHz (in the 
opposite connection). One way of achieving this conversion is to convert 
the signal to analog form and then resample it at the new sampling rate. 
This approach is susceptible to electronic noise in the analog circuitry 
and is limited by the dynamic range of the analog components. 

A more attractive approach to the sampling rate conversion process 
is to do a direct digital-to-digital conversion of the sampling rate. This 
conversion can be done as accurately as desired and is not prone to ex- 
traneous noise from electronic components. The digital-to-digital con- 
version is accomplished with the aid of a linear phase FIR digital inter- 
polating filter whose output sample values are computed at a different 
sampling rate than the incoming samples.’ 

Figure 5a shows the frequency response of a 119-tap FIR lowpass filter 
which was used in the 10 kHz to 16 kHz conversion. Although the length 
of the filter is 119 samples, only 15 multiplications and additions per 
output sample are required in the conversion process because only a 
subset of the filter coefficients are needed in computing each output 
sample.’ Similarly, Fig. 5b shows the frequency response of a 127-tap 
linear phase FIR filter used in the 16 kHz to 10 kHz sampling rate con- 
version. In this case 26 multiplies and adds are used in computing each 
output sample. 
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Fig. 4—(a) Log magnitude response. (b) group delay, and (c) impulse response of 
bandpass filters. 


ill. FACTORS AFFECTING THE TANDEM LINK 


The performance of the LPC to CVSD link is affected by several pa- 
rameters. Since the LPC vocoder analyzes and then synthesizes the 
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Fig. 5—Frequency response of (a) 119-tap FIR filter for 10:16 sampling rate conversion 
and (b) 127-tap FIR filter for 16:10 sampling rate conversion. 


speech signal, the performance of the CVSD coder will be affected by the 
manner in which the speech waveform is synthesized. Primarily the 
performance of the CVSD coder can be affected by factors such as the 
input level of the speech and the peak factor (ratio of peak-to-RMS value) 
of the synthesized waveform. Alternatively, parameters in the narrow- 
band system relating to the pitch and the coefficients of the all-pole filter 
in the synthesis model have little bearing on the performance of the CVSD 
coder. Therefore, our investigation of the LPC to CVSD link concentrates 
primarily on the first effects (input level and peak factor). 

The input level of the speech waveform determines the operating mode 
of the CVSD coder. If the input level is too low the coder will be operating 
in the region in which its performance is determined primarily by 
granular noise. If the input level is too high the coder will operate ina 
slope overload condition. Typical waveforms for these coder conditions 
are shown in Figs. 6-8. Figure 6 shows a complete sequence of waveforms 
for the wideband system in Fig. 2 under normal (maximum SNR) oper- 
ating conditions. Figure 6a shows 100 msec of speech appearing at the 
output of the 10 kHz to 16 kHz sampling rate converter. In Fig. 6b the 
speech waveform has passed through the first bandpass filter (see Fig. 
2) and the effects of bandlimiting and phase distortion can be observed. 
Figure 6c shows the output waveform of the CVSD coder with the gain 
G = 0.158 which results in maximum SNR. The effects of quantization 
are clearly noticeable. Finally, Fig. 6d shows the CVSD coder output after 
bandpass filtering (i.e., the output of the wideband system). Figure 7 
shows waveforms for the coder operating in the granular noise region 
(G < 0.158). In Fig. 7a and 7b, waveforms of the unfiltered and band- 
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Fig. 6—Speech waveforms for the wideband system. (a) Waveform after 10:16 sampling 
rate conversion. (b) Waveform after first BPF (input to CVSD). (c) CVSD output waveform 
(G = 0.158). (d) CVSD output waveform after BP filtering (output of wideband system). 


pass-filtered CVSD coder output are shown for a gain setting of G = 
.009375 or about 25 dB below the maximum SNR operating point. The 
effects of severe distortion are clearly visible and the speech was com- 
pletely unintelligible at this point. In Fig. 7c and 7d, waveforms are 
shown for unfiltered and filtered coder outputs with G = .0395 or about 
12 dB below the maximum SNR operating point. Figure 8 shows examples 
of waveforms for the coder operating in the slope overload region (G > 
0.158). In Fig. 8a and b the unfiltered and bandpass-filtered CVSD coder 
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Fig. 7—Output waveforms of the CVSD coder in the granular noise region. (a) Coder 
output for G = 0.009375 and (b) the output after BP filtering. (c) Coder output for G = 
0. 0395 and (d) the same output after BP filtering. 


output is shown for G = 2.528 or about 24 dB above the maximum SNR 
operating point. Although the effects of severe slope overload are ap- 
parent, the intelligibility of the coder in the slope overload region is not 
greatly reduced from that at the maximum SNR. Finally, Fig. 8c and Fig. 
d show unfiltered and filtered output waveforms of the CVSD coder for 
G = 0.632 or about 12 dB above the maximum SNR operating point. 
One measure of coder performance is signal to quantizing noise ratio 
(SNR). The range of input signal level over which the coder maintains 
an acceptable SNR is often used as a measure of the dynamic range of 
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Fig. 8—Output waveforms of the CVSD coder in the slope overload region. (a) Coder 
output for G = 2.528 and (b) the output after BP filtering. (c) Coder output for G = 0.632 
and (d) BP filtered output. 


the coder. The point of optimum SNR Is achieved when the coder is on 
the verge of slope overload.® Unfortunately this operating point is not 
the same as the optimum operating point observed on the basis of sub- 
jective performance.® Subjectively the noise due to slope overload is less 
objectionable than the granular noise. Therefore, SNR by itself is not a 
reliable means for determining the optimum operating region of the 
coder. More will be said about this in the next section, and in Part 2 (the 
accompanying paper) another measure of coder performance is proposed 
which correlates better with subjective performance than the SNR 
measure. 

An important factor affecting the performance of the CVSD, at least 
in terms of its SNR, is the peak factor of the LPC synthesized speech. The 
step size of the coder tends to track the RMS level of the input and, if the 
speech waveform has a large peak-to-RMS ratio, slope overload will cause 
the peaks to be clipped giving the speech a hoarse sound. If the clipping 
is severe, intelligibility is degraded. 

The peak factor of the synthesized speech can be reduced in several 
ways to make it more amenable to waveform coding. In one technique 
the standard pitch source excitation to the LPC synthesizer (an impulse), 
is modified to spread the energy of the pitch pulse over a larger portion 
of the pitch period.’ A pulse which is spread over about 7 percent of the 
pitch period has been found to be effective for this purpose.? Two pulse 
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Fig. 9—Pulse excitation sources used for the LPC synthesizer. (a) Rectangular pulse 
shape. (b) Rounded pulse shape. 


shapes were tried in this experiment—a rounded pulse shape and a 
rectangular pulse shape. The rectangular pulse shape is shown in Fig. 
9a. The energy in the pulse is normalized to that of an equivalent im- 
pulse. The second pulse shape, shown in Fig. 9b, is a rounded shape 
proposed by Rosenberg® (pulse shape B) to approximate the shape of 
an actual glottal pulse. T,, is defined as the opening time and Ty is de- 
fined as the closing time of the pulse. The pulse shape is then defined 
by the relation 


F(t) =B |3 (=) -2 G)| for 0<t <T> 


ie ji 
ane 
F(t)=B as ree ) for Ty <t < Tp + Tn (15) 
N 


where F(t) is the height of the pulse and B is its peak amplitude. Values 
of T, and Ty used in the experiment are T,, = 0.05T and Ty = 0.02T 
where T'’ is the pitch period. The width of the pulse therefore expands 
or contracts dynamically with the pitch period. The rounded pulse shape 
was found to give the most natural sound for the LPC synthesized 
speech.? 

A second technique that can be used to reduce the peak factor of the 
LPC sythesized speech is to filter the speech with an allpass filter which 
disperses the energy of pitch peaks in the waveform. One approach to 
designing such an allpass filter has been proposed by Rabiner and Cro- 
chiere!® in which the parameters of an allpass filter were optimized to 
spread the energy of an impulse signal under the limitations of a maxi- 
mum peak amplitude. This allpass network has been effective in re- 
ducing the peak factor of the LPC synthesized speech. 

The allpass filter which was used in our experiments was an eighth- 
order filter which was cascaded three times to give a total allpass filtering 
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_ Fig. 10—Group delay of the allpass filter used for preprocessing the CvsSD coder 
input. 


equivalent to that of a 24th order filter. The z-transform of each 
eighth-order filter is of the form 


H(z) = tl H(z) (16) 


where 
b; _ cz +272 


17 
1 — cjz71 + b;z-2 ( ) 


H(z) = 


and the coefficients are 


b; = 0.8149 c, = 1.2308 
bo = —0.4970 co = —0.1060 
b3 = 0.8621 c3 = —0.2185 
b4 = 0.7870 c4 = 1.5727 


The total group delay of the 24th order all-pass filter is given in Fig. 10. 
It is seen that the group delay is dispersed between 5 and 90 samples (0.5 
to 9 msec) across the frequency band (0 to 5 kHz). 

Fig. 11 shows the effects of pitch pulse modifications and allpass fil- 
tering on a voiced region of speech. Figure 11a shows the natural speech 
waveform and Fig. 11b shows an equivalent section of LPC synthesized 
speech using an impulse excitation. In Fig. 11c and d waveforms are given 
for LPC synthesized speech with the rectangular and rounded pulse ex- 
citations respetively. Figure 11e shows the waveform for the LPC impulse 
excited speech which was allpass filtered. Finally, Fig. 11f and g show 
the combination of both allpass filtering and rectangular and rounded 
source excitations respectively. It is seen that the rectangular or rounded 
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(g) 
Fig. 11—Waveforms of the LPC synthesized 5 eens (a) Natural speech input. (b) LPC 


synthesized speech with impulse excitation. (c) LPC synthesized speech with rounded 
source excitation. (d) LPC synthesized speech with rectangular source excitation. (e) Allpass 
filtered waveform of LPC speech with impulse excitation. (f) Allpass filtered waveform 
of LPC speech with rounded source excitation. (g) Allpass filtered waveform of LPC speech 
with rectangular source excitation. 


excitation source modifications do improve the peak factor of the speech 
as does the allpass filtering. The combination gives a further improve- 
ment. In the next sections we investigate the effects of these modifica- 
tions on the performance of the CVSD system. 


IV. SNR MEASUREMENTS OF THE CVSD SYSTEM 


In this section we report on the performance of the CVSD coder in the 
tandem link as a function of the signal gain and the modifications of the 
peak factor of the LPC synthesized speech. Computer simulations were 
made for the system shown in Fig. 2. Two sentences were used for the 
simulations. The first sentence, “Every salt breeze comes from the sea,” 
was spoken by a low-pitched male and was recorded off a conventional 
telephone line. The second sentence, “I know when my lawyer is due,” 
was spoken by another male into a high-quality microphone. 

The signal-to-quantizing noise ratio (SNR) of the CVSD coder was 
measured across the entire sentence. The CVSD noise was obtained by 
subtracting the filtered output from the CVSD input (also filtered) as 
shown in Fig. 2. The gain G of the signal was varied from 0.009375 to 
2.528 or over a range of approximately 50 dB. 

Table I shows the resulting SNRs for the first sentence, “Every salt 
breeze ....”” Column 1 corresponds to results for natural speech input 
to the CVSD coder. Columns 2, 3, and 4 are for LPC synthesized speech 
using an impulse source, a rounded source, and a rectangular source 
excitation respectively. Table II gives corresponding SNR’s measured 
with the all-pass filter preceding the CVSD. Tables III and IV pertain 


NARROWBAND-TO-WIDEBAND LINK 1717 


Table | — Snr of CvsD coder vs. gain and source excitation 


Coder SNR* (dB) 


LPC synthesized speech 


Gain Original Impulse Rounded Rectangular 
G speech source source source 
0.009375 2.00 1.84 1.77 1.78 
0.0395 7.30 7.35 7.89 
0.158 9.29 8.89 10.47 10.62 
0.316 7.29 7.23 9.00 
0.632 5.06 5.13 6.39 6.76 
1.264 3.31 3.34 4.17 4.43 
* For sentence “Every salt breeze comes from the sea.” 
Table Il — SNR of CvsbD coder vs. gain and source excitation for 


allpass filtered inputs 


Coder SNR* (dB) 


LPC synthesized speech 


Gain Original Impulse Rounded Rectangular 

G speech source source source 
0.009375 1.57 1.63 1.74 1.84 
0.0395 7.53 7.51 7.91 7.94 
0.158 9.26 9.67 10.33 10.79 
0.316 7.95 8.37 9.41 9.68 
0.632 5.83 6.10 7.18 7.53 
1.264 3.60 3.84 4.68 4,92 
2.528 2.00 2.15 2.63 2.76 

* For sentence “Every salt breeze comes from the sea.” 

Table Ill — SNR of CVSD coder vs. gain and source excitation 


Gain 


0.009375 


0.0395 
0.158 
0.316 
0.632 
1.264 
2.528 


Original 
speech 


2.52 
8.93 
11.14 
9.48 
7.07 
4.50 
2.02 


Impulse 
source 


2.37 
8.80 
10.77 
8.90 
6.61 
4.38 
2.64 


Coder SNR* (dB) 


LPC synthesized speech 


Rounded 
source 


2.28 
8.85 
11.61 
10.01 
7.54 
4.96 
2.95 


* For sentence “I know when my lawyer is due.” 


Rectangular 
source 


2.31 
9.06 
12.01 
10.46 
7.81 
5.10 
3.03 


to the sentence “I know when ...” and show measurements corre- 
sponding to those in Tables I and II, respectively. 

The data indicate that, with or without the allpass filter, CVSD SNR 
with natural speech input is quite similar to SNR with speech derived 
from an LPC synthesizer with impulse excitation. (In all four tables the 
greatest difference between an entry in Column I and the corresponding 
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Table IV — SNR of CVSD coder vs. gain and source excitation for 
allpass filtered inputs 


Coder SNR* (dB) 


LPC synthesized speech 


Gain Original Impulse Rounded Rectangular 

speech source source source 
0.009375 1.98 1.42 1.69 1.67 
0.0395 8.33 8.48 8.91 9.05 
0.158 10.59 10.44 11.67 11.49 
0.316 9.53 9.02 10.42 10.16 
0.632 7.22 6.89 7.84 7.86 
1.264 4.59 4.76 5.44 5.55 
2.528 2.54 2.75 3.20 3.31 


* For sentence “I know when my lawyer is due.” 


entry in Column 2 is 0.6 dB; most differences are less than 0.4 dB.) 
Comparing Columns 3 and 4 with Column 2 in the tables we see that 
broadened pitch pulses lead to 1-2 dB improvements in measured CVSD 
performance in the slope overload region. As a rule the rectangular pulses 
result in a slightly higher SNR than rounded ones. 

The benefits of allpass filtering are less pronounced than the benefits 
of broadened pitch pulses. Comparing Column 2 entries (impulse exci- 
tation) in Table I and Table II, we see that the allpass filter offers im- 
provements of about 1 dB in SNR at high levels for one sentence. Tables 
III and IV show virtually no improvement with the other sentence. When 
the synthesizer uses broadened pitch pulses (Columns 3 and 4) the all- 
pass filter adds 0.5 to 1 dB to CVSD performance with the first sentence 
and little or nothing to the SNR of the second sentence. 

Figure 12 displays the range of possible improvements in CVSD SNR 
relative to the conventional tandem configuration which includes an LPC 
synthesizer with an impulse source and no allpass filter at the narrow- 
band-wideband interface. The lower curve in Fig. 12a and b shows CVSD 
for this configuration for the two sentences in our study. The upper curve 
in Fig. 12a pertains to the most successful modification of the sentence 
recorded form a telephone line. This modification involves rectangular 
pitch pulses and an allpass filter. With the sentence recorded from a 
high-quality microphone, the best SNR performance, plotted in Fig. 12b, 
was obtained with the rectangular excitation and no allpass filter. 


V. SPEECH QUALITY 

Informal judgments of the processed speech suggest that the pre- 
dominant distortions of tandem circuits are those of CVSD. However, 
the quality of a vocoder such as LPC depends on speaker and utterance 
while a waveform coder such as CVSD is relatively insensitive to speech 
material. Although the utterances used in this work were amenable to 
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SENCENCE: “EVERY SALT BREEZE....” 


A IMPULSE SOURCE 
© RECTANGULAR SOURCE 


ALLPASS FILTERED 


SNR IN DECIBELS 
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INPUT LEVEL, dB RELATIVE TO G = 0.158 


(a) 


Fig. 12—Summary of the results of the SNR measurements of the CVSD coder, for sen- 
ae (a) “Every salt breeze comes from the sea” and (b) “I know when my lawyer is 
ue”. 


LPC, we anticipate that for certain speakers LPC would be the weaker 
link in a tandem connection. 

As a function of input level, CVSD quality appears to be much lower 
with weak inputs, which lead to substantial granular quantizing noise, 
than with strong inputs, for which the main distortion is slope overload. 
This subjective effect is at variance with SNR indications which show 
rapidly declining quality as the input level rises into the coder overload 
range. 

The use of broadened LPC excitation pulses lends a more natural 
quality to the resynthesized speech as well as improving CVSD SNR in 
the overload region. An allpass filter which also improves SNR for one 
sentence seems to offer little, if any, enhancement of subjective quality 
of tandem circuits. 


Vi. DISCUSSION 


Although the conclusions of the previous section must be regarded 
as tentative, pending formal subjective evaluation of speech processed 
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SENTENCE: “t KNOW WHEN MY LAWYER IS DUE.” 
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O RECTANGULAR SOURCE 
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(b) 
Fig. 12 (continued) 


in tandem connections, it does appear that efforts to improve the quality 
of the wideband link would be justified. The CVSD encoder is a 9-year-old 
design with values of circuit elements chosen to withstand transmission 
errors occurring at rates as high as 10 percent. If this very demanding 
requirement is relaxed somewhat and recent advances in delta modu- 
lation are incorporated, it may be possible to modify the CVSD to produce 
higher stand-alone and tandem quality. Alternatively other 16 kb/s 
wideband coding schemes such as adaptive PCM, adaptive differential 
PCM or sub-band coding may offer even greater advantages than im- 
proved Cvsp.®1! 
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In this paper the tandem link of a 16 kb/s Continuously Variable 
Slope Delta modulator (CVSD) waveform coder and a 2.4 kb/s Linear 
Predictive Coding (LPC) vocoder is studied. Of prime concern are the 
effects of the CVSD coder on the LPC vocoder analyzer. In particular 
the problems involved in making a reliable voiced-unvoiced decision, 
estimating pitch period, and estimating LPC coefficients from the coder 
output are studied. It is shown that LPC coefficient estimation from 
the CVSD output is highly inaccurate. An analytical distortion measure 
(an LPC distance) is used to show the magnitude of the distortion in- 
troduced by the coder as a function of the signal gain into the CVSD 
coder. Although the remainder of the LPC analysis (i.e., pitch detection, 
volced-unvoiced decision, and gain calculation) can be performed 
reasonably accurately, the magnitude of the distortions in estimating 
the LPC coefficients is sufficiently large to make the vocoded speech 
barely intelligible and of poor quality. 


I. OVERVIEW OF THE TANDEM LINK OF CVSD TO LPC 


In the first part of this paper we discussed the effects of the narrow- 
band system (the LPC vocoder operating at 2400 b/s) on the wideband 
system (the CVSD waveform coder).! There it was shown that one of the 
major issues was tailoring the signal characteristics of the vocoded speech 
to reduce the peak factor, thereby reducing the amount of slope overload 
noise generated in the CVSD. When we consider the tandem link of CVSD 
and LPC, far more serious problems are encountered since we must es- 
timate the basic speech production parameters (i.e., pitch, voiced-un- 
voiced, LPC coefficients) from a severely degraded signal. Since speech 
parameter estimation is an imperfect process, even on high-quality 
speech, the effects of the CVSD coder, which include quantization noise 
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Fig. 1—Block diagram of signal processing operations in tandem link of a CVSD coder 
and an LPC vocoder. 
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as well as slope overload noise, could potentially make the tandem link 
totally unacceptable. 

In this paper we discuss several aspects of a tandem link consisting 
of a CVSD waveform coder, and an LPC vocoder. Our purpose is to dem- 
onstrate the range of signal levels over which the LPC can operate rea- 
sonably well in tandem with the CvSD coder. Figure 1 shows a block di- 
agram of the signal processing used in implementing and testing a 
CVSD-LPC tandem link. The speech signal s(n) is assumed to be sampled 
at a 10-kHz rate. Thus the first block in Fig. 1 is an interpolator to raise 
the sampling rate of the signal to 16 kHz. The interpolator described in 
Part 1 of this paper was used here.! The 16-kHz signal was then sharply 
bandpass-filtered from 200 Hz to 3200 Hz using the 8th-order elliptic 
bandpass filter described in Part 1 of this paper.! To simulate variations 
in overall signal level into the CVSD coder, a variable gain G was applied 
to the filtered 16-kHz signal. The gain G was varied from 0.009375 to 2.5 
in the simulations which gave about a 50-dB variation in signal level over 
which the system was studied. To compensate for the input scaling, a 
gain of 1/G was used at the output of the CVSD coder. The output of the 
coder was again sharply bandpass-filtered from 200 to 3200 Hz to remove 
the wideband quantization noise generated in the CVSD coder. For 
compatibility with the LPC system the signal was then decimated toa 
10-kHz sampling rate using the decimator described in Part 1 of this 
paper. 

Figure 2 shows a block diagram of the processing required for the LPC 
vocoder. The LPC analyzer estimates the following control parame- 
ters: 


(i) Pitch period 

(it) Voiced-unvoiced decision 
(iit) Signal gain 

(iv) LPC parameters 
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Fig. 2—Block diagram of LPC analyzer and synthesizer. 


The LPC synthesizer uses the estimated parameters to recreate the 
speech in the manner shown in Fig. 2. The details of the analysis and 
synthesis methods are described in Part 1 of this paper. 

Based on our knowledge of both the techniques used in LPC analysis 
and the degradations introduced by the CVSD coder, it was anticipated 
that the voiced-unvoiced decision and the LPC parameter estimation 
algorithms would be most affected by the CVSD coder. Thus, in the next 
two sections we discuss the specific algorithms used for voiced-unvoiced 
detection (along with pitch detection) and show results on how the al- 
gorithms performed in the tandem link as a function of the signal level 
into the CVSD coder. In Section IV we present results on the accuracy 
with which the LPC parameters were estimated from the coder output. 
For a measure of similarity between coder input and output, the LPC 
distance measure proposed by Itakura is used. Finally, in Section V we 
discuss the interactions between the CVSD coder and the LPC vocoder 
and suggest some possible ways to improve the performance of a tandem 
link of a wideband and a narrowband system. 


ll. PITCH DETECTOR AND VOICED-UNVOICED DETECTOR USED IN 
THE TANDEM LINK 

As discussed in the preceding section, the choice of an appropriate 
pitch detector and voiced-unvoiced detector is critical to the proper 
operation of the LPC vocoder. Based on a series of intensive investigations 
into both objective and subjective rankings of a variety of pitch detec- 
tors,2-3 it was shown that simple waveform pitch detectors would be in- 
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Fig. 3—Block diagram of modified autocorrelation pitch detector. 


adequate for a severely degraded waveform such as obtained at the 
output of a CVSD coder. Thus either a sophisticated correlation-type 
pitch detector, or a spectral-type pitch detector is required for this ap- 
plication. From this class of pitch detectors both the AMDF* and 
AUTOC® pitch detectors were found to be moderately fast, and suffi- 
ciently robust over a wide variety of transmission conditions and pitch 
range of the speaker. Because of the familiarity of the authors with the 
AUTOC pitch detector, this method was finally selected. 

Before the method of operation of this pitch detector is reviewed, some’ 
comments must be made about the selection of the voiced-unvoiced 
detector. Ideally one would prefer to make a voiced-unvoiced decision 
prior to, and independent of, the pitch detection. In this manner the role 
of the pitch detector is strictly to make the best estimate of pitch period, 
given a priori that the segment is accurately classified as voiced. For 
unvoiced segments, the pitch detector is not used at all. There have been 
at least three proposed methods for making a voiced-unvoiced decision 
prior to and independent of any pitch detection.*8 However, all three 
methods suffer from the necessity of having a training set of data that 
characterizes the signal classes. For CVSD coding, the variability of the 
signals due to variations in gain is exceedingly large—i.e., a 40-dB vari- 
ation in input level can change the signal from one with a large amount 
of granular noise to one with a large amount of slope overload noise. 
Therefore, making a voiced-unvoiced decision accurately without a 
periodicity measurement (pitch detector) to aid the decision is extremely 
difficult. Thus, the voiced-unvoiced decision is combined with the pitch 
detection in the AUTOC method. 

A block diagram of the AUTOC pitch detector is given in Fig. 3. The 
method requires that the speech be lowpass-filtered to 900 Hz. Thus a 
99-point linear phase, FIR digital filter is used here.? The lowpass-filtered 
speech is sectioned into overlapping 30-msec (300 samples at 10 kHz) 
sections for processing. Since the pitch period computation for all pitch 
detectors is performed 100 times/second—i.e., every 10 msec—adjacent 
sections overlap by 20 msec or 200 samples. 

The first stage of processing is the computation of a clipping level Cy 
for the current 30-ms section of speech. The clipping level is set at a value 
which is 64 percent of the smaller of peak absolute sample values in the 
first and last 10-ms portions of the section. Following the determination 
of the clipping level, the 30-ms section of speech is center clipped, and 
then infinite-peak-clipped, resulting in a signal which assumes one of 
three possible values, 1 if the sample exceeds the positive clipping level, 
—1 if the sample falls below the negative clipping level, and 0 other- 
wise. 

Following clipping, the autocorrelation function for the 30-ms section 
is computed over a range of lags from 20 samples to 200 samples (i.e., 
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Fig. 4—Block diagram of system used to compare pitch contours from two pitch de- 
tectors and to perform an appropriate error analysis. 


2-msec to 20-msec period). Additionally, the autocorrelation at 0 delay 
is computed for appropriate normalization purposes. The autocorrelation 
function is then searched for its maximum (normalized) value. If the 
maximum (normalized value) exceeds 0.25, the section is classified as 
voiced and the location of the maximum is the pitch period. Otherwise, 
the section is classified as unvoiced. 

In addition to the voiced-unvoiced classification based on the auto- 
correlation function, a preliminary test is carried out on each section of 
speech to determine if the peak signal amplitude within the section is 
sufficiently large to warrant the pitch computation. If the peak signal 
level within the section is below a threshold computed from the back- 
ground noise level, the section is classified as unvoiced (silence) and no 
pitch computations are made. 


ill. EFFECTS OF CVSD CODING ON PITCH DETECTION 


To investigate the effects of CVSD coding on pitch detection, two 
sentences were used whose pitch contours were known extremely accu- 
rately. Figure 4 shows a block diagram of the experimental arrangement 
used to show pitch detection errors in the tandem link. The speech, s(n), 
is analyzed by the SAPD method? to give the reference pitch contour, 
Dr(m), m = 1, 2,..., M, where M is the number of 10-msec frames in 
the utterance, and p,(m) = 0 if the frame is classified as unvoiced. 
Otherwise p,(m) is the estimated pitch period. Extensive tests have 
shown the SAPD method to be a reliable and robust procedure for ob- 
taining the reference pitch contour.? 

The test pitch contours are obtained by sending the speech either 
directly to the pitch detector, or first through the CVSD coder where the 
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signal level is determined by the gain G. We denote the test pitch contour 
as p:(m), m = 1, 2,..., M. The error analysis compares p,(m) and p;(m) 
over the utterance and makes the following measurements: 


(i) Average pitch period error during voiced regions, P, defined 
as 


es M 
P=—- 5 [p-(m) ~ pelm)] (1) 
vm=l1 
pr(m)#0 
pt(m) 40 
[pe(m)—p;(m)|<10 


where N, is the number of voiced regions satisfying the conditions that 
the reference pitch contour indicates a voiced region (p,(m) + 0), the 
test pitch contour indicates a voiced region (p;(m) ~ 0), and the dif- 
ference in estimated pitch period is less than or equal to 10 samples 


(|pz(m) — p,(m)| < 10). 
(it) Standard deviation of the pitch period during voiced regions, op, 
defined as 


HS Gina (2) 
ro = | 57%, elm) ~ pel)? — P| 


Pr(m)#0 
Ppt(m)~0 
| pt(m)—pr(m)| <10 


(iii) Number of voiced-to-unvoiced errors, N,,, defined as 
M 
Nou = a g(p,(m), Pi(m)) (3) 


where 
g(x,y) =1 if x>Oand y=0 
= 0 otherwise (4) 


(iv) Number of unvoiced-to-voiced errors, N,,,, defined as 
M 
Nu = ¥, &(Pe(m), pr(m)) (5) 
(v) Number of gross pitch period errors, Ng, defined as 
M 
Ne = 2X f(p,(m), pe(m)) (6) 
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Table | — Error analysis for utterance ‘‘Every salt breeze comes 
from the sea’ 


(a) Analysis on raw pitch data 


Signal P op Nou Nuv Ne 
Original speech 0.142 —-0.786 8 7 1 
CVSD-G = 0.009375 1.154 1.925 69 73 29 
CVSD-G = 0.0395 0.221 0.901 22 18 6 
CVSD-G = 0.158 0.252 0.874 7 8 4 
CVSD-G = 0.316 0.288 0.961 6 8 5 
CVSD-G = 0.632 0.294 0.952 3 12 4 
CVSD-G = 1.264 0.397 1.037 5 23 4 
CVSD-G = 2.528 0.397 1.159 4 37 10 

(b) Analysis on nonlinearly smoothed pitch data 

Signal P op Nou Nuv Nog 
Original speech 0.156 0.756 7 1 0 
CVSD-G = 0.009375 1.589 1.236 91 24 1] 
CVSD-G = 0.0395 0.556 1.029 15 2 0 
CVSD-G = 0.158 0.426 1.0738 7 0 0 
CVSD-G = 0.316 0.282 0.922 6 0 0 
CVSD-G = 0.632 0.356 0.920 2 1 0 
CVSD-G = 1.264 0.367 1.155 1 4 0 
CVSD-G = 2.528 0.490 1.253 0 6 1 
where 

f(pr(m), pe(m)) = 1 if p,(m) # 0, p.(m) # 0, 
|[p-(m) — pz(m)| > 10 
=0 otherwise (7) 


Since many of the errors made in pitch detection are easily corrected 
by a nonlinear median-type smoother,!° the test arrangement in Fig. 4 
also shows the capability of passing both the reference and test pitch 
contours through such a smoother prior to the error analysis. Results 
will be presented on both the raw data and the smoothed data. 

Results obtained on two different sentences are presented in Tables 
I and II, and some of the key results are summarized in Figs. 5-8. Ut- 
terance 1 was the sentence “Every salt breeze comes from the sea” spoken 
by a low-pitched male and recorded off a conventional telephone line. 
The utterance had 256 frames (i.e., it was 2.56 seconds long), of which 
108 were unvoiced and 148 were voiced. Table I shows values of P, Op; 
Nou, Nuv, and Ng as a function of the gain G, for both the raw data and 
the nonlinearly smoothed pitch contours. Figure 5 shows plots of Nou 
versus G (plotted in dB on a normalized scale) for both the raw and 
smoothed data, and Fig. 6 shows plots of N,,, versus G. Results obtained 
on the original utterance (uncoded) are also presented as a means of 
comparison. 

As seen in Table I, values of P for the coded speech were about 2 to 
3 times larger than for the original speech (except for G = 0.009375). 


1730 THE BELL SYSTEM TECHNICAL JOURNAL, NOVEMBER 1977 


Table I! — Error analysis for utterance ‘‘l know when my lawyer 
is due”’ 


(a) Analysis on raw pitch data 


Signal P op Nou Nuv Ne 
Original speech 0.304 0.796 uf 3 0 
CVSD-G = 0.009375 0.192 2.722 21 12 63 
CVSD-G = 0.0395 0.304 0.738 17 2 7 
CVSD-G = 0.158 0.193 0.660 10 1 2 
CVSD-G = 0.316 0.209 0.639 10 1 4 
CVSD-G = 0.632 0.228 0.812 9 2 4 
CVSD-G = 1.264 0.225 0.922 6 3 5 
CVSD-G = 2.528 0.221 0.993 8 4 9 

(b) Analysis on nonlinearly smoothed pitch data 

Signal P op Nou Nuv Nog 
Original speech 0.323 0.617 1 2 0 
CVSD-G = 0.009375 1.247 2.922 25 10 40 
CVSD-G = 0.0395 0.382 0.656 18 1 0 
CVSD-G = 0.158 0.172 0.573 11 1 0 
CVSD-G = 0.316 0.213 0.549 12 1 0 
CVSD-G = 0.632 0.252 0.711 11 1 0 
CVSD-G = 1.264 0.257 0.823 10 0 0 
CVSD-G = 2.528 0.329 0.985 10 0 0 


However, values of P were all less than 0.5 samples (except for G = 
0.009375) indicating that the average pitch period errors, due to the 
coder, were still relatively insignificant. For a gain of G = 0.009375 (large 
amounts of granular noise) the pitch detection process broke down en- 
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Fig. 5—Plot of number of voiced-to-unvoiced errors versus CVSD signal level for utter- 
ance “Every salt breeze comes from the sea.” 
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Fig. 6—Plot of number of unvoiced-to-voiced errors versus CVSD signal level for utter- 
ance “Every salt breeze comes from the sea.” 


tirely. Thus, at this extreme the LPC vocoder cannot possibly operate. 
However, as was shown previously, for this value of gain the CVSD coder 
produced unintelligible speech; hence we need not be concerned with 
this result. 

Values for o, for the coded speech were essentially identical to those 
obtained for the original utterance. Also the number of gross pitch period 
errors was small for all values of G except G = 2.528 and G = 0.009375, 
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Fig. 7—Plot of number of voiced-to-unvoiced errors versus CVSD signal level for utter- 
ance “I know when my lawyer is due.” 
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Fig. 8—Plot of number of unvoiced-to-voiced errors versus CVSD signal level for ut- 
terance “I know when my lawyer is due.” 


and all these errors were correctable by the nonlinear smoother, as shown 
in Table Ib. Thus, one can conclude that for cases in which both the 
reference and test pitch contours were classified as voiced, the coder did 
not impede accurate determination of the pitch period—i.e., pitch is well 
preserved in the CVSD output. 

Now the major question is how well the voiced-unvoiced decision could 
be made on the coder output. An examination of Table I and Figs. 5 and 
6 shows that, for several values of G, a substantial number of un- 
voiced-to-voiced errors occurred. However most of these errors were 
easily correctable by the nonlinear smoother since the estimated pitch 
periods (when such errors occur) are essentially random, and are auto- 
matically “smoothed” to zero (i.e., unvoiced). Also some of the voiced- 
to-unvoiced errors are corrected by the smoother. 

For this sentence it is concluded that over a fairly large variation in 
coder input gain, the deterioration of the signal is not so large so as to 
make pitch detection unreliable. 

A second set of results is given for the utterance “I know when my 
lawyer is due” spoken by another male speaker over a high-quality mi- 
crophone. This sentence had 175 frames (1.75 seconds) of which only 13 
were unvoiced and 162 were voiced. Thus this utterance was essentially 
all voiced. Results obtained on this utterance are given in Table IJ and 
Figs. 7 and 8. Again it is seen that, except for G = 0.009375, values of P, 
gp, and Ng (smoothed) are essentially the same for the coder output as 
for the original. Since there were very few unvoiced frames, the number 
of unvoiced-to-voiced errors is also the same for the coded speech as for 
the original. However, the number of voiced-to-unvoiced errors for the 
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coded speech is much larger than for the original speech. Most of these 
errors occur in the region of the /z/ in “is due,” and as such are not cor- 
rectable by the nonlinear smoother. However, the errors in this low- 
intensity region are not very preceptible and therefore such errors are 
not overly crucial. 

In summary we have shown that the CVSD coder preserves the pitch 
of the speech over a reasonably large signal range and that the voice- 
unvoiced decision can also be reliably made over a fairly large dynamic 
range of coder inputs. 


IV. EFFECTS OF CVSD CODING OF ESTIMATION OF LPC 
COEFFICIENTS 

The next issue to consider is the effects of the CVSD coder on the es- 
timation of the LPC parameters. The LPC coefficients model the com- 
bined transfer function of the vocal tract, glottal source, and radiation 
load. Incorrect estimates of the coefficients can seriously perturb the 
frequency spectrum of the modeled speech signal and, hence, affect the 
intelligibility of the synthesized sound.!! 


4.1 Distance measure 


To evaluate objectively the spectral distortion introduced by the CVSD 
coder, an LPC distance measure proposed by Itakura was employed. 
The LPC distance measure is defined as 


t 
d, = log aca 


b, Vbt (8) 


where 


an = LPC coefficient vector (1, a1, ...,a@,) measured in the nth frame 
of the original uncoded speech signal. 
bn = LPC coefficient vector measured in the nth frame of the CVSD 
coded speech signal 
and V is the speech correlation matrix with elements V;; defined as 


N-|i-j| 
Vij = v(ji —j|) = 5 x(n)x(n + |i —Jj]) (9) 
i= 
where x(7) is the speech signal and N is the number of samples in the 
frame. 

Figure 9 shows examples which illustrate how the measured d, is 
useful in measuring the degree of spectral deviation of the coded sound 
from that of the original.* Although the measure d, is not the only 
possible indicator of spectral distortion,!* it has been shown to closely 


* The quantitative significance of d,, is discussed in detail in Ref. 14. 
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Fig. 9—Plots of typical spectra and the resulting values of d, for three examples. 


correspond to perceptual judgments.!4 In addition, the measure has been 
effectively applied in problems of speech recognition,!? speaker recog- 
nition,!> and variable frame rate synthesis. Before discussing the results 
of the LPC distance evaluation of the CVSD coder, it is important to 
emphasize that d,, is not a perfect measure of perceptual changes in the 
character of the sound.!!!7 However, it is a good measure of spectral 
deviations, which is a useful indicator of intelligibility loss.'4 


4.2 Evaluation 


The two sentences utilized in the investigation of pitch detection ac- 
curacy were also employed in the evaluation of the effects of CVSD dis- 
tortion on the estimation of the LPC coefficients. For each sentence, the 
LPC coefficients for the uncoded, original speech are first calculated. The 
LPC parameters are calculated 50 times per second at a uniform rate 
using the autocorrelation method!8 with a 30-msec Hamming window. 
The speech is preemphasized using a first order digital network with 
transfer function 


H(z) = 1-—0.952z7} (10) 
prior to LPC analysis in order to minimize the effects of performing the 
LPC analysis at a uniform rate (i.e., pitch asychronously).!9 The results 


of this analysis provide the reference LPC coefficients (the a,,’s) for each 
20-msec frame. 
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SENTENCE: EVERY SALT BREEZE COMES FROM THE SEA 


G = 0.009375 





FRAME NUMBER 


——— "“PERCEPTUALLY” SIGNIFICANT BOUNDARY 


Fig. 10—Values of d, versus frame number as a function of CVSD signal level for ut- 
terance “Every salt breeze comes from the sea.” 


A similar LPC analysis is performed for each of the various CVSD coded 
versions of the original sentences. These analyses provide the b,,’s for 
use in the calculation of distance (d,) between the original sentence and 
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SENTENCE: | KNOW WHEN MY LAWYER IS DUE 


—— — “PERCEPTUALLY” G = 0.009375 
SIGNIFICANT BOUNDARY 





qn 


dn 





FRAME NUMBER 


Fig. 11—Values of d, versus frame number as a function of CVSD signal level for ut- 
terance “I know when my lawyer is due.” 


the particular CVSD-coded sentence. Figures 10 and 11 show the 
frame-by-frame LPC distance measured for each CVSD-coded version 
of the two original sentences. The dashed line in the figures refers to a 
suggested threshold of d,, = 0.9 for a just-perceptible difference.!4 Figure 
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“EVERY SALT BREEZE...” 





0 
0.009375 0.158 0.632 2.528 
GAIN (G) 


Fig. 12—Plots of average LPC distance (d) as a function of CVSD signal level (G) for both 
test sentences. 


12 shows the average LPC distance as a function of gain G. The average 
distance is defined as 


(a= a 11 

u a n (11) 
where M is the number of frames in the sentence. 

The results of the LPC distance analyses are striking in that the dis- 
tance uniformly decreases as the gain G increases. This result is in direct 
opposition to the SNR findings discussed in the first part of this paper.! 
According to the LPC distance measure, the CVSD-coded sentence is 
improving in quality (i.e., closer in distance to the original) as the gain 
increases. However, according to the SNR measurements, the similarity 
between the original and the CvSD-coded sentence is decreasing as the 
gain increases beyond G = 0.158. Although the dissimilarity between 
the waveforms of the original and the CVSD-coded version with G = 1.264 
is apparent from Fig. 13, it is interesting to note that informal perceptual 
experiments indicate that the quality of the CVSD coder is actually im- 
proving as the gain G increases. Since the LPC distance measure is sen- 
sitive to spectral distortions, it is (in this case) a better measure of quality 
than SNR. The use of the LPC distance measure as an indication of speech 
quality has been suggested by other authors.!4 


V. COMPATIBILITY OF CVSD WITH LPC 


As a final check on the performance of the entire system, an informal 
perceptual evaluation of the CVSD-LPC tandem link depicted in Fig. 1 
was performed. The LPC vocoder was efficiently designed for a bit rate 
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(a) ORIGINAL WAVEFORM 





(b) CVSD—CODED WAVEFORM (G = 1.264) 


Fig. 13—Waveform pie of one section of an utterance and the resulting output of the 
CVSD coder for G = 1.264. 


of 2.4 kb/s2° and the CVSD was designed for 16 kb/s operation using the 
various gains G. For the smallest gain, G = 0.009375, the speech was 
unintelligible. For the higher gains, the output speech was intelligible, 
but the quality was significantly worse than the quality of the 2.4 kb/s 
LPC synthesis. The quality of the tandem link appeared to saturate (or 
even become slightly worse due to the poorer estimates of pitch and gain) 
for G = 0.158. Even for the best-quality output, the combination of CVSD 
noise and the parametric distortions of the LPC vocoder rendered the 
tandem a marginal communications link. 


VI. SUMMARY 


In the tandem link of a wideband and narrowband speech communi- 
cation system in which the wideband system was a 16 kb/s CVSD coder 
and the narrowband system was a 2.4 kb/s LPC vocoder, the CVSD coder 
was shown to be the weak link. The major distortion introduced by the 
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CVSD coder was spectral distortion as measured using an appropriate 
LPC distance measure. This distortion was sufficiently severe to make 
the LPC output, although intelligible, of poor quality. It was further 
shown that the waveform distortion in the CVSD coder was not so severe 
so as to make pitch detection unreliable, and even a reliable voiced- 
unvoiced decision could be made on the CVSD-coded speech. 

The major conclusion from this study is that alternative 16-kb/s coders 
be considered as the wideband communication system for such com- 
munication links. Possible alternatives include ADPCM systems,”! sub- 
band coders,22 and transform coders.23 
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Discrete-Time Single Server Queues with 
Correlated Inputs 
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A wide variety of queueing systems with a single server can be mod- 
eled by the equation bn+1 = (bn — 1)*+ + Zn, where b, denotes queue 
length and z, the input. The usual assumption about the sequence {zp} 
is that it be a sequence of independent identically distributed (1. 1. d.) 
random variables. However, in many applications, this is not really the 
case; {z,}is a sequence of correlated random variables. We show that 
with the help of a transformation, a (k + 1)-dimensional Markov pro- 
cess that suffices to describe the queueing system may be found, where 
k is the memory of the input process. We derive an equation for the 
steady-state generating function corresponding to the joint distribution 
of this vector process. We find that a simple set of equations can be 
obtained for the marginal distributions. In particular, the steady-state 
distribution of bn, the queue length, can be obtained without solving 
for the joint distribution. 


|. INTRODUCTION 


Several computer systems and networks involve queueing models with 
single server queues. We consider a discrete-time queueing system, with 
service time normalized to unity, modeled by the equation 


bn41 = bn, — 142, if b, 2 1 
= 21 if b, = 0 
or equivalently 
bnt1 = (bn —1)t + 2p (1) 
Here b,, denotes queue length! and the nonnegative integer valued se- 
quence z,, is the input. 


A vast majority of literature in queueing theory deals with the case 
when {z,,} is a sequence of independent identically distributed random 
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variables. In this situation, when the average value Ez, < 1,6 = limn}.* 
b,, is a well-defined random variable, and various authors have analyzed 
the distribution of b; see Ref. 1. 

An interesting approach is due to Spitzer? who uses a simple conse- 
quence of eq. (1): when bo = 0 then 


: 

bras = max | ¥ 2-17} (2) 
r 1=0 

to derive an integral equation for the distribution of b. However, we will 

follow the approach that models {b,,} as a Markov process as in Ref. 3. 

Here the theory of Markov chains can be used to derive formulas for the 

equilibrium distribution of b,, that is, the distribution of b. 

The literature dealing with models where {z,,} are not necessarily in- 
dependent is relatively scant. Recently Ali Khan* and Herbert® have 
analyzed the case when z,, is the state of adenumerable Markov chain. 
In this case (6,,,2,,) forms a Markov process, thus relaxing somewhat the 
condition that {z,,} are independent identically distributed (i. i. d.) ran- 
dom variables. 

The queueing process that motivated the work presented in this paper 
arose in a data communications system. Messages are temporarily stored 
in a buffer before they are sent across the communications network. It 
is assumed that the buffer transmits one packet, the basic unit of data, 
in a unit time interval, provided that it is not empty. In this context, then, 
Zn is the number of packets that arrive at the buffer in the time interval 
(n,n + 1]. It is assumed that the inputs are correlated and z,, is taken to 
be asum of moving averages. 

In order to illustrate the techniques, the particular example z, = x} 
+ x}_, + x2 is first analyzed. This corresponds to the arrival of two kinds 
of messages. The first kind of message consists of two packets which are 
spread apart in time, the second packet being transmitted two units of 
time after the first packet. The number of such messages generated in 
the (n + 1)st time unit is denoted by x}. The second kind of message 
consists of just one packet, and the number of such messages generated 
in the (n + 1)st time unit is denoted by x2. It is assumed that (x},x2), 
n=0,1,2,..., are independent identically distributed vector random 
variables. However, for each n, x} and x2 may be dependent. In partic- 
ular, if . 

E(ty**to*») = @[(1 — p)ty + pto] 


with 0 < p < 1 fixed, then the probability that a message is of the first 
kind is 1—p, and the probability that it is of the second kind is p. 


* We mean here limit in distribution: for each J, 


lim Pr {bn Sj} =Pr{b <j} 
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There are several other examples where such a model for the input 
process 2, is more appropriate than the usual one. We give two examples. 
Consider a queueing system where each request for service may consist 
of a sequence of tasks to be completed by the same server. However, these 
tasks may not be available for completion in the same time interval; in- 
stead they are spread out in time. Hence the random variables corre- 
sponding to the number of tasks arriving at the server may be correlated 
as in the above example. This model may apply to a scheduler in a 
computer processing system. Another example, that of a dam fed by 
rivers that originate at geographically distant points, motivated the 
model considered by Herbert.® When rainfall occurs, affecting the flow 
in all of the rivers, the increase in flow to the dam is spread out in time 
since the origins of the rivers are at different distances from the dam. 
A discrete time model of the dam process, similar to the one in the packet 
network example above, can be solved by the method presented in this 
paper. 

In general we assume that 

ar 
Zn = o” ’ OFS aj (3) 
1=1 j=0 

where the nonnegative integer valued random variables in the sequence 
{(xn,X%, ... ,%,)} are independent and identically distributed, and a are 
nonnegative integers with a} > 0 for each i. For each n the random 
variables x},x2,...,x«/ may be dependent on each other. Notice that z, 
by itself is not necessarily a Markov process. As far as we know there is 
only one work dealing with a special case of eq. (3) which is related to 
ours. Herbert® considers the case when 


k 
Zn Ps QiXn-j (4) 

j=0 
where {x,} are i. i. d. random variables and a; are positive integers. In 
this case whenever x, ~ 0, bn+; 40,1 =1,...,k +1, hence b,+, is lin- 
early related to 6,41, r = 2,...,k + 2 from eq. (1). From this property, 
formulas can be derived for the equilibrium distribution for b, given 
Xn—-1, Xn—2,---» Xn—k. However, even in this special case our approach 
gives formulas for 

b= lim b,, 
nto 

itself more simply than the method of Ref. 6. 

In the general case b,, is not a Markov process, but it is shown that, 
with the help of a transformation, a (k + 1)-dimensional Markov process 
that suffices to describe the queueing system may be found. The first 
component of this Markov process is just b,. An equation is derived for 
the steady-state generating function corresponding to the joint distri- 
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butions. This equation involves a multinomial, which corresponds to zero 
queue length. It is shown that a finite system of linear equations can be 
obtained to solve for the coefficients in this multinomial. A simple set 
of equations for the marginal distributions is then derived, leading to 
the calculation of the steady-state generating function of the queue 
length. 

In Sec. II we review the case when z, arei. i. d. random variables. An 
example for a system where z,, is a moving average is worked out in Sec. 
III to illustrate our method. In Sec. IV we introduce the model considered 
in this paper and describe the transformation that leads to the simpli- 
fication in the solution. The generating function of the underlying vector 
Markov process is derived in Sec. V. The method of solving for certain 
parameters that occur in Sec. V is described in Sec. VI. The isolation of 
marginals and the derivation of a simple set of equations for them is the 
subject of Sec. VII. A pair of limiting cases of the input process is ana- 
lyzed in Sec. VIII. Finally, for a special class of problems, some formulas 
relating the limiting cases are also derived in Sec. VIII. The terminology 
of Markov chains used in this paper is consistent with that of Ref. 3. 


ll. QUEUE WITH INDEPENDENT INPUTS 


When {z,,} is a sequence of independent identically distributed random 
variables, it follows that b,, is a Markov process. The number of packets 
waiting to be transmitted, b,, serves as the state for a Markov chain S. 
The state space of S is the set of nonnegative integers. The transition 
probabilities for S are generated by eq. (1) as follows: 


P™*) 4 Pr {by =i} = = {bn41 = i[b, = J} PP 
J2 


= 2 Prien a OS) 45) 


jz0 
Let Pr {z, = i} = p; fori = 0,1,.... Then, since z, is a nonnegative in- 
teger, 
a i+1 
PP" = piPo + x Pi-j+1P} (6) 
= 


When 1 > po > 0,8 is irreducible and aperiodic. The following theorem 
gives conditions under which S is positive recurrent. 


Theorem 1: The Markov-chain S is positive recurrent when Ez, < 1. 


When S is positive recurrent then 


lim 6, = b 


nto 
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is a well-defined random variable and the equilibrium distribution of 
by, that is, the distribution of b, is such that 


lim Pr {b, = j} = aj > 0 forj =0,1,... (7) 


nto 


Furthermore, if P? = 7;,7 = 0,1,...,sois Peet and 7; are the unique 
nonnegative solution to the infinite system of linear equations: 


mj =1 


ime 


j 
and, fori =0,1,..., 
i+1 
Ti = Dito + Xu Pi-j417j (8) 
= 
These are obtained from eq. (6) by substituting P? = P?*! = 7;. 


For a proof of the above results see Karlin.? 

In order to solve for the equilibrium distribution we will employ the 
method of generating functions. For any random variable x, the gener- 
ating function of x, ¢,(s), is defined as 


¢x(s) = Es*, |s| <1 (9) 
Let 
gn(s) = Es'n = + PP s!, 
i=0 


Then using eq. (1) and the independence of b,,z, we have 


Es bati = Es (bn—1)t Egs2n (10) 
From the definition of ¢,, it follows that 
bn.1(s) = (s~!@n(s) + (1 — s!)P§)62(s) (11) 


where ¢z(s) = Es?n. Assuming that Ez, <1and1> po > 0, let the gen- 
erating function of 


b = lim b,, 


nto 


be 
o(s) = > ms! 
[see eq. (7)]. Then from above it is clear that if ,(s) = $(s) then ¢n41(s) 


= ¢(s). So from eq. (11) we get 


(1 — s—+)z0¢-(s) 


eT = 514,00) 


(12) 
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To find zo we take expectations of both sides of eq. (1) and take the limit 
asn {| , Then 
To = = Ez, (13) 
So 
(1 — s)¢2(s)(1 — Ez) 
bz(s) — s 


This gives the generating function of b in terms of ¢,(s). However, to 
get 7;, we need not invert the generating function ¢(s). Treating ¢,, ¢ 
as formal power series, using II; to denote LiLo 7;, and equating like 
powers of s in eq. (14), we can show: 


Ilo = 79 = 1— Ezy 


o(s) = (14) 


Il, = (opi + To — pillo)/po 


J 
0 = (mop; + j-1- a pill) [Po (15) 
| ed 


Equations (15) give explicitly the formulas needed to solve for x; or 
II;. Notice that any finite number of the 7;’s can be determined by 
solving a finite number of linear equations. Informally we refer to such 
a situation as being finitely solvable. 


Il. AN EXAMPLE OF A QUEUING PROCESS WITH CORRELATED 
INPUTS 

In the context of the application discussed in 1, there are instances 
when the data arriving at the buffer form a sequence of correlated ran- 
dom variables. For an example we consider here a case when there are 
two classes of sources that generate data. The first kind generates two 
packets whenever it transmits a message. However, these packets are 
not generated simultaneously; instead they are spread apart in time, the 
second packet being transmitted two seconds after the first one. The 
number of such messages generated in the (n + 1)st second is denoted 
by x}. The second class of sources generates messages of one packet each 
and the number of such messages generated in the (n + 1)st second is 
denoted by x2. (x1,x2),n =0,1,2,..., are assumed to be independent 
identically distributed vector random variables. Note that, for each n, 
x} and x2 may be dependent. Then the number of packets arriving at 
the buffer in the (n + 1)st second is 


Zn =xitxi +x? (16) 


So the number of packets in the buffer at the end of the (n + 1)st second 
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is given as in eq. (1) by 
bn+1 > (bn = Lyra ee ee (17) 


It is clear b,, is not a Markov process. However, (6, x}_1, x}, x21, x25) 
is a five-dimensional Markov process. We will derive another Markov 
process from eq. (17) that is only three-dimensional and suffices to de- 
scribe the queueing process. Define 


Yon = bn 
Yin = Yon + Xn-2 
Yon = Yin + Xn-1 (18) 
Then from eq. (17) we have 
Vone = (On = 1)" = Youl tia ten tah 
vient = [Gon Dt = Yon) + yea Fen P25 
yan+i = [yon — 1)* — yon] + yon + 2x5 + x5 (19) 


Let Von = Vin = x} + x2 and von = 2x} + x2. Then (Von, Vin, Von) is 
independent of (yoj, y1;, ¥2;) for j < n by assumptions about x}, x2. 
Hence (yon, Yin) Yon) is a three-dimensional Markov process. The state 
space of the corresponding Markov-chain S can naturally be indexed 
by a triple of nonnegative integers. Let 


Pi iio = Pr {yon = to, Yin = 11, Yon = tol (20) 
Then 


nt+1. a: _: dee 
Piinie = : os Pr {¥on41 = 10, Y1n+1 = 11, Yon+1 = bol 
JOJ1J2 


Yon = Jo Yin =J1 Yan =JAP Ri jo (21) 


These form the equations for transition probabilities. Notice that not 
all states (i9,11,12) communicate with (0, 0, 0). For example, we can show 
that when iy = 0, the only states that communicate with (0, 0, 0) are (0, 
0, 0) and (0, 1, 1). Suppose yon+1 = 0, Yine1 = U1 and yons1 = le. Then 
bn+1 = 0. Hence, from eq. (17), b, < 1,x), = 0. But x} = 0 implies yon41 
= Yin+1- Also, b, < 1 implies x},_; < 1. Further, yon+1 = 0and x}_, < 
1 imply yin+1 < 1. However, it can be shown that states that do not 
communicate with (0, 0, 0) are transient (see Sec. IV). So we will restrict 
the state space by allowing it to consist only of those states, denoted by 
A, that communicate with zero. We will continue to denote by S the 
Markov chain on the restricted state space . Then S is irreducible and 
aperiodic (see Sec. IV). Notice that for every state at time n 


Yon S Yin S Yan (22) 


QUEUES WITH CORRELATED INPUTS 1749 


Later in this paper we will show that S is positive recurrent when Ez,, 
< 1. For now we will assume this is so. Interpreting the sums over Jo, J1, 
jg to extend only over 4 we have from eqs. (19) and (21), and the defi- 
nitions of Von, Vin, Ven, 


Pi aie p24 . Pr Von = 19 mee Von = lg — ja} PO jnie 
J1—lOF=J2711 
+ me Pr {von = 1+ 19 — Ji, Van = 1 +12 — ja} Ph jsjo (23) 
jo>0,j1-i0=j2-i1 
The equilibrium distribution of S: 


: n —_ Go- OS dy, 
lim Peis - Piojis,i2 
nto 


has the property that if P®, ;, ;. = Pioi,i. for (io, 11, i2) € A, so does PR ,.. 
So Pip,i1,i9 satisfies: 


= P 48 : 3a : 

Pigs So a : Pr Von = 19 — J1, Van = le — jo} Posi 
J1—l0F=J2711 

+ ae Pr {von = 1 + 19 — Ji, Van 


Jo>0,J1—t0=Je-11 
=1+ lo a Jo} Pigias (24) 
Pio,is,ig = 1 


(i0,t1,12)€A 


P,,i1,i9 18 the unique nonnegative solution of eq. (24) (see Ref. 3). In 
principle, solving the infinite system of linear eq. (24) determines P;,i, io; 
hence the equilibrium distribution of (von, Yin, Yon). However we will 
see a much simpler way to find equilibrium distributions of the com- 
ponents Yon, Yin, Yon, Without computing P;,;,,1.. Denote Es" by ni(s) 
and Es’ by ¢iv(s). Then from eq. (19) we can derive the following 
equations paralleling eq. (11): 

én+1,0(s) = [s~!bni(s) + (1 — sein(s)]bou(s) 

énti,1(8) = [s~@no(s) + (1 — s~4)ean(s)] O10 (s) 

ént1,2(s) = [s~!dno(s) + (1 — s~)eon(s)] G00 (s) (25) 
Here 


Cin(s) = a Pr {Yon = 0, Vin = j}s/, 1= 1,2 (26) 
jz 


For any n the only admissable states in 4 that have yo, = 0 are (0, 0, 0) 
and (0, 1, 1). So ¢;,(s), i = 1,2 are polynomials of degree 1, and c1,,(s) = 
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Con(s). Let 6;(s) denote the generating function of 
yi = lim yin 
nfo 
and 
cj (s) = mn Cin(S) 

fort = 1,2. Then 

go(s) = [s~1¢r(s) + (1 — sex(s)] bou(s) 

b1(s) = [s~ Gols) + (1 — s~)eq(s)] or (s) 

pas) = [s~*o(s) + (1 — s~)ex(s)] G20 (s) (27) 
From eq. (27) 


(1 — s~e1(s) dbo, (s) 
1 — s~!$(s) 


Since doy, ¢1y, 62, are known directly from the distribution of x}, x2, 
eq. (27) gives ¢0,¢1 in terms of c;(s), the only unknown. Let c,(s) = Ro 
+ k,s. Then 


€3(1) = Pr {yo = 0, v1 = 0} + Pr {yo = 0, y1 = 1} 
= Pr {yo = O} = ko + ky 


2(s) = (28) 


As in eq. (13) 
kot ky =1— E(x} 4+x1_,4+ x2) 
=1-kz, (29) 
In order to derive another equation for ko,k; we go back to the original 
equations for Pipi ,i., eq. (24). From eq. (24) we can derive the following: 
for ig = 11 = lz = 0, since v;, are nonnegative, Poo9 = Pr {Von = 0, Ven = 


0} Pooo + Pr {von = 0, Van = 0} P11. However, von = 2x} +x? = 0 implies 
x} = x2 = 0,s0 Vo, = 0. Therefore 


Ro = Pooo = Pr {von = 0} (Pooo + Pits) (30) 
Similarly 
ky = Pou = Pr (von = 0} Piz 
Pig = Pr (von = 1, Von = 2} Pia 
+ Pr {von = 1, Von = 2} Pooo 
Hence 


ky = Pr {von = 0} Pr {von = 1, von = 2} (Pooo + Piss) (31) 
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Notice that the various probabilities occurring on the right-hand sides 
of eqs. (30), (31) can be calculated from the distribution of (x1,x2). For 
example: 


Pr {von = 1, Van = 2} = Pr {x} = 1, x2 = 0} 


Therefore using eq. (29) we can determine ko,k1, hence c1(s). From eq. 
(27), therefore, it is easy to derive the formula for ¢o(s), namely 


-2 


+ S—!1y(S)dov(s) + fools) | 


Ss Ioiy (s) | 
1 — s~4$9,(s) 

To solve for the equilibrium distribution of b,, i.e., distribution of yo, 
we do not have to invert ¢o(s). It turns out that eq. (27) can be translated 
to linear recursions for marginal distributions for yo, v1, Yo. Hence, as 
in Sec. II, the distributions of yo, yi, ye are finitely solvable. That this 
is so, in the general case, is shown in Sec. VII. 


= (se) dulls) E + (32) 


IV. QUEUEING PROCESSES WITH MOVING AVERAGE INPUTS 


The most general input process that we will consider in this paper is 
a finite sum of moving averages, i.e., 


=> sl aixt_; (33) 


i=1lj= 
Equation (1) in this setting is 


é ok . 
bn+1 = (bp — 1)* + pp te ae (34) 


The integer k is referred to as memory of the input process z,. Under 
the assumptions below, the (k¢ + 1) dimensional vector process (bn, x}_1, 
xii, . 62, Kip, 02-1, --+, Xen. +, MEL, ---, £6-2) is Markov as in the 
example of Sec. III. 

However, by a transformation we will find a (k + 1) dimensional 
Markov process that suffices to describe the queueing system. Define: 


on = bn 
and, forr =0,1,...,k —1, 
4 k oy As 
Yr+1,n = Yrn + 2 ; >; ay Lnajty (35) 
i=1j=rt+1 
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Let 


are 
MX = Urn 


iM 


r : z 
> aj = wand 
J=0 i 


forr =0,1,...,. Then using eq. (34) we can verify: 
Yr,n+1 = [(yon = 1)t = Yon] + Yr+1,n + Urn, T= 0, 1, tne Rd (36) 


Yk,nt+1 = [(yon — 1)* Yon] + Ykn + Ven (37) 


We make the following assumptions for the rest of this paper: The a! 
are assumed to be nonnegative integers and, for eachi, a4 > 0. We will 
assume that the vector, nonnegative integer valued random variables 
(x7, x2,...,x4) are independent and identically distributed, though for 
each n, x}, x2,...,x< will be allowed to be dependent on each other. We 
will also assume that Pr {v, = 0} > 0 and Pr {v,, > 1} > 0 for some r. 

From the assumptions about x4, (Von, Vin. +» Ukn)! = Vp is inde- 
pendent of y; = (yoj, y1j,---» Yej)’ for] <n. Hence y, isa (k + 1) di- 
mensional Markov process. The state space corresponding to this 
Markov process is indexed naturally by a (k + 1) triple of nonnegative 
integers. Furthermore by definition of y;,, 1 = 0,1,...,R,n =0,1,2,..., 
Yon < Yin S Yon S++* S Yen. Hence we can assume that if (io, 11, .. . , iz) 
denotes a state then 


io Sly Slg See Sik (38) 


Let A’ denote the set of vectors satisfying (38) and S’ the Markov chain 
with state space 4’. Of the states in 4’ let A denote the set of states that 
communicate with the state 0 = (0,0,..., 0)*. Using the following the- 
orem, we will be able to restrict our attention to only those states that 
are in A, and to the irreducible Markov chain S, with state space 4, 
derived from S’. 


Theorem 2: 

(t) Every state in 4’ of the form (m, m,...,m)* belongs to A. 

(1) Every state of S’ transitions to a state belonging to A in at most 
k steps. 

(iit) Every state in .4 is accessible from a state of the form (m,m,..., 
m)! in at most k steps. 

(iv) Sis irreducible and aperiodic. 

(v) For each ig, the number of states in A which are of the form (io, 
11,.--, lp )* is finite. 


Proof: Let F denote the (k + 1) X (k + 1) matrix with elements F;;41 = 
1,fori =0,...,R —1, Fp = 1, and Fj; = 0 otherwise. Also, let 1 denote 
the vector (1, 1,...,1)*. We note that F1 = 1 and, for any y = (yo, y1,.-., 
yp)', Fy = (yp, .--5 Veo Vk» » ++» VR)* by induction. Equations (36) and 
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(37) can then be written in vector form as follows, using o, to denote yon, 
— (Yon — 1)*. Note that o, = 1 if yo, > 0, and o, = 0 if yo, = 0. 


Ynt1i = Fy, — on 1+ vp, n = 0,1, 2,... 


We can then show that forn -12120 


= prmiy, "5 gL +S Pe-tiy; (39) 

j=l j=l 
Hence, if v; = O0forj =n —1,...,n —k, then it follows from (39), with 
i=n-—k, thaty, isa vector of the form m 1 = (m,m,...,m)' for some 


nonnegative integer m. Therefore, since Pr {v; = 0} > 0 by assumption, 
0 is accessible from any state by allowing v; to be zero for as many con- 
secutive j’s as needed. We assumed earlier that Pr {v,,, > 1} > 0 for some 
r. Hence, since Vpn = Urn, Pr {vzj = M} > 0 for some integer M > 1 and 
all j. Therefore, if yo = 0, vz; = M forj =0,1,...,n — 1implies yz, > 
nM — n. Hence Pr {yzn > nM — n, yo = 0} > 0. For every sequence v; 
such that v; = 0 forj=n,n+1,...,n+k,..., Yn+n4; remains pro- 
portional to 1 for all i = 0. Therefore, for each m, Pr {yn+x4; =m 1, yo 
= 0} is greater than zero for some n,i dependent on m. From (39) we can 
therefore show that any state of the form m 1 communicates with 0 and 
hence belongs to A. 

If yo ¢ A’, then we will prove, irrespective of what v;’s are for j = 0, 
1,...,2 —1, that y; ¢ A, by showing that yz, is accessible from a state 
of the form m 1 in at most k steps, where 


k 
m=k+ yro- » oh-j (40) 
J= 


Let Yj, j =0,1,..., be the sequence of states traced by S’ if yo is set 
to zero but v;,j =0,1...,% — Lare left unchanged. If oj = yo; — (oj — 
1)*+ for] = 0,1, 2... then (39) holds with primes on y;’s and o;’s, and 
Yo = 0. We will first prove that for each k, y, = y;, by showing that for 
each 1: 
y= 9: 20,Foi—y¥) —Gi-y¥) =0 
= Vier — Vier 2 0, F(yita — Viti) — (Viti — Yin) 20 (41) 

Suppose the assumptions i in (41) hold as they do fori = 0. Then y;; — 
Vi = VR-Li = pe. heat yop Yoi 2 0. Therefore if Yoi > Voi then 
(yi; — yj) + (6; — 6;) 1 = 0, which is trivially so if yo; = yo;. Hence, using 
(39) and corresponding equations for y;41, 

Viti — Vier = Fly; — yj) + (6; — o)1 2 (y; — y;) + (6; — o)1 20 
Furthermore 

F(yi+1 - Yi+1) Aye = Yi+1) = F[F(y; - y;) = (yp = y;)] 20 


1754 THE BELL SYSTEM TECHNICAL JOURNAL, NOVEMBER 1977 


since all elements of F are nonnegative. Hence in particular, 
ko k 
Vi Ver Jot Do Ope 2 op = 0 
J=1 j=1 
Therefore from (40), with 


k 
s= ¥ op-,5+mZk. 
jel 
Now let y; be the sequence of states traced by S’ if yp were set to m1 
while v;, j = 0,1,...,& — 1 were left unchanged. Then, with o; = yo; — 
(yor — Lt, 


Yi -Yi= (m + 2 Oj; ee oj yl (42) 


for each 1. As before, , using (41) we can ae y¥;-y;20. From (42) we 
have Viti — Yn= am Yi - yi a (o; — ai )1, hence yi = ve =< oj a 0; 7 Viti 

= yi, If <7 = 0 for some1 < k, then Yoi = 0, but Yor 2 Yop hence yo; = 
yoj and y; = y; from (42).S0 0; =0=> y,, = y,, fork =n = i. So in par- 
ticular 


uw t k ‘ k Ww 
Yi Va= (m+ YD oh-j— Y oh-j)1=0 (43) 
j=1 j=l 
However, we noted earlier that 


stm= m+ ¥ one) >k 
jz 


Hence (43) can only hold if 


. k 
stm=k= > o4-; 
j=l 
Therefore we have shown that for eachi < k, o; = 1. Now we use eas. (40) 
and (41) to show 


; i i 
Yr — Yr = (M— ypzo)l - pz o,—j1 + a op-j1= 0 (44) 
jJ=1 j=l 


Since y;, belongs to 4, being accessible from m1 belonging to A, we 
have shown that, starting from any state in 4’, S’ transitions into a state 
in A in at most k steps. Furthermore, every state of .4 is accessible from 
some state of the form m1 in at most k steps. It is clear from the defini- 
tion of .4 that S is irreducible. To show that S is aperiodic, we merely 
note that y; can equal zero for arbitrarily many consecutive j’s with 
positive probability. 

We will now prove that the set of states in 4 which are of the form (0, 
11,-..,U4)¢ is finite. This result is used later to derive conditions for the 
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positive recurrence of S. We just showed that every state in A is acces- 
sible from a state of the form m1 in k steps. In particular, if a state of the 
form (0, i1,...,%%)£ is yz, with yo = m1 for some m then 


k k 
Yor =m — 9) on-j + X Up—j,j-1 = 0 
l= 


j=l 
k k 
Yrr =m — Do op-j + DY UR j-1 (45) 
j=l j=l 
Hence 
k 
m y= OR-j <0 
j=l 
and 
k e ok 
» Ve-j,j-1 = x > My—-j X5-1 <k 
J=1 1=1j=1 
Therefore 
e 
oy 1G xia) Sk 
i=l = 
since, fori = 1,2,..., ¢, aj = ui = 1 and ty > ph forj =0,1,...,&. 


Therefore, from eq. (45), 


é 
2 " k (46) 
i=1 
Hence for every state 
é. 
(0,71,..-,tr)’e A,ip SRY ui, 
i=1 

hence such states are finite in number from eq. (38). In a similar way we 
can show for any integer j the states (Uo, 11,..., 1%) « A such that ig S 
j is a finite set. 

The transition probabilities for S can be derived from eqs. (36) and 
(37). Let P? = Pr lyon = 10, Yin = 11, -- » Yen = ty}. Then 


PE =  Priyner = il¥n = HPS (47) 
je 
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PRY = Y Pr (von = io — Ju. + +s Vk—-1n = tk-1 — Fhe Ven 
Jo=0 
jot 
= in —Je}P§ + YO Pr (von =to~ Jit .,...,Ve-1n 
Jo>0 
jeA 
= ip-1—Jr+1,U0pn =ie —Jn +1} P? (48) 
If the equilibrium probabilities 


P; = lim P® 
nto 


exist, then P? = P; for every ic A implies P?*! = P;, ie A. Furthermore 
P; is the unique nonnegative solution of 


ve ey eae! 
icA 


P,= » Pio—jiit—joputk—fee 3 
Jo=0,jeA 


+ OD Pig fit it—jottnie—iet1Pj (49) 
Jo>0,jeA 
Here pi,iy,...i, = Pr {Von = to, .. Ven = ir}. 
We show next that S is positive recurrent when Ez,, < 1. 


Theorem 3: S is positive recurrent if 
Bon ae ok 
ED pix <i 
i=1 
Proof: Define a new process c,, as follows: 


4 a 
Cn+1 = (Cn = 1) e D, Hi Xy 
t=1 


=(c, —1)+ + vpn (50) 


We know that the Markov chain corresponding to c,, is positive recurrent 
if Evz, <1 from Theorem 1. In particular if Ev;, < 1 then 


lim Pr {c, = 0} >0 
nto 
The processes c,, b, as defined by eqs. (50), (34) are related by x!, i = 
1,...,@. Let bg = 0 and let r be such that b,-; > Ofori =0,1,...,r—- 
1 and b,,_, = 0. Then eq. (84) implies that 
r £ Rk 
bn+1 = a 2 


a eee —r (51) 
m=0 0 


J= 
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We also know from eq. (2), assuming cp = 0, that 

rtk @ | 

Cn41= 2D, Dd BeXn-m — (r +k) 

m=0i=1 

From the definition of }, it can be easily verified that 
Cnt1 2 On41—k 
Hence 
Pr {bn+1 Sk} = Pr {en+1 = 0} 

When Evu;, < 1 we know from Theorem 1 that 


lim Pr {c,+1 = 0} >0 
nfo 


hence 
ws inf Pr {b,41 < k} >0 
Let the set of states (Zo, 11,..., ig) in 4 withio < k be denoted by Ap. 


Let P? be the probability of S being in state i at time n starting from j 
at time 0. Then, we have shown that 


lim inf >° P{&>0 


nfo 1c A 
Since cardinality of A, is finite 
lim inf P%, > 0 
nfo 
for some ie A;. We also know that 0 is accessible from i. So P; > 0 for 
some r. Therefore 


lim inf Pat" > lim inf Pr, P% > 0 


nto nfo 


Hence 0 is positive recurrent. Therefore, since S is irreducible and 
aperiodic, S is positive recurrent. 


V. GENERATING FUNCTIONS FOR JOINT DISTRIBUTIONS 


We will now derive expressions for joint distributions of (yo, y1,..., 
yr) assuming Ez, < 1,80 S is positive recurrent. Let 


k 
E (1 sy) = $n (80, 81,--- Sk) 


and 


k 
J (IIs?) = oy(So, $1,---, 8k), |Si] <1 
r= 
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From eqs. (36) and (37) we have, using independence of v,, and yp, 


dn+1(So, $1, -- +» Sk) 


k (yon—1)t—yon k-1 
= E (( Il sr) syrthn st") 
r=0 


r=0 
x dy (So, S1,.005 Sr) (52) 
Proceeding as in eq. (11) we can show 


k 
dn+1(S0, $1,+++» Sk) = | én(t, SQ, $1, +++, Sk—2 Sk—-18k) II sj? 


l= 


k 
+ (1 _ I] 37") $n (0, So, S1,---; 54180) | 
i=0 


Xx by (So, S1,.--,5 Sr) (53) 
When ¢, is the generating function of the equilibrium distribution Le., 
when 
k ri 
on (So, S1,--+-, Sk) = (so, St,-+-,5 Sr) = EK I si (54) 
= 
then ¢,+41 = ¢. Therefore ¢ satisfies 


k 
(so, S1,.-+55 Sp) = | ou, So, S1,--+, Sk—2, Sk—1Sk) II sy? 


l= 


k 
+ (1 = it 371) (0, So, $1,---, Sk—2, 51-180) | 
l= 


x by (So, $1, uae: J Sr) (55) 


We note that ¢(0, t1,..., ¢,) is a polynomial of finite degree since the 
set of states (0,71,...,2,) is finite. Knowledge of (0, t1,... , t,) deter- 
mines (So, $1,..- , Sz) as follows. If we set so = 51 =-+-+ =Sz-1 = 1 then 
(55) becomes 


é(1, eee » 1, Sp) = [sz’ p(1, Dy atag ly S) 
7 (1 = sp')o(0, 1, Li say i sp) }ov (1, 1, ne) 1, sz) 


This determines ¢(1, 1,..., 1, s,) in terms of ¢(0,1,1,...,1, sz): 
~(1, 1, cee 1, Sr) 


= (1 a si )(0,1,1, Sears d, Sr) ov (1, i ae 81g ds Sk) 


56 
1—s;' gy (1, Law lyse) { 


Forr =0,1,...,R set 
p' (sr, ae Sk) = ¢(1,1, e989 1, Sr, Sr+iy++ +5 Sr) (57) 
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Then eq. (56) determines ¢* in terms of ¢(0, 1,...,1, s,). Using eq. (55) 
yields: 


k 
FC | 1 Pr. | ee ae ee 
=r 


k 
+ (1 =] sj") (0, ib saisdrssuceita)| 
i=r 


K Gols Laws despises gp) (58) 


So starting with ¢*, k applications of (58) yields #%(so,..., Ss) = 
(So, ..-, Sz) in terms of 4(0, 51,..., Sz). Equations (15) have a coun- 
terpart here. These can be derived in mechanical fashion using formal 
power series expressions for ¢ and ¢,. we will not go into the details here. 
The derivation is analogous to that given in Sec. VII for the margin- 
als. 

In the case ¢ = 1, an alternate generating function was considered in 
Ref. 8. The corresponding generating function is obtained by setting 


k 
uj = [[5,,j7=0,...,Rk 
i=j 
and defining 
$(so, $1,.--, Se) = B(Uo, Uy,..., UR) 


Rk 
_ lim E (usm Il uyero-tn) 


n—->o r=1 


Then corresponding to eq. (55), 


P(Uo, Uy,..., UR) = [UQ’ (Uo, Uo, U1, ..-, Up-1) 
+ (1 a Up‘) &(0, Uo, U1,..-; Up—1)] By (Uo, Uj,..-, Up) (55’) 


where 


> 


P, (Uo, Miys:20 UB) =E( 


usrn) 
0 


r 


with 


é xe 
Wrn = 2) Xn 
i=1 


It follows from eq. (55) that, for j = 0,1,...,k — 2, (k = 2), 


&(s, --+,8,U],..., Up—j) am [s!&(s, -++,5,U],.-.-; UR—j~1) 
+ (1 = s—1)&(0, S,...,5,U4,..., Up—j—1)] 
X @,(s,...,5,U1,...,UR-j) (58’) 
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and 


P(s,..., 8, Uz) 
= [s~!@(s,...,5) + (1 —s7)8(0,s,...,s)]®,(s,...,5,uy) (58”) 


These equations are equivalent to eq. (58). If we set u; = s in (58”) then 
we may solve for ®(s,...,s) = @(1,...,1,8), as in (56). 


VI. FINDING $(0, $1,..., Sx) 


We will show here that a finite system of linear equations can be ob- 
tained to solve for the coefficients of the polynomial of finite degree that 
represents (0, si,..., Sp). Let 


R 2 
Yj(s1,---, Se) =u II s?! 
i=1 


where p = Pr{yo = 0} and let 6;(so, s1,..., sz) be related to yj as (so, 
S1,...,8p) Is to f(0, 51, ..., Sz) in eq. (58). That is, if 05 is defined as in 
(57), 03 (s;,..., Sr) = 6; (1, 1,...,1,58,, S-41,..., Se), then 6% satisfies 
the set of equations equivalent to (58): forr =0,1,...,k—1 


k 
0% (S;,..-, 8k) = bi 871 05! (s,,..., Sk—2, Sh—-18R) 


=r 


Rk 
+ (1 = I] 37") ¥,(1, il, cers i, Sr, +++, Sk-2; 51-181) | 
i=r 


X Pp (1, 1,..-51,8,,...,8%) (59) 


and (56) corresponds to 


(1 — 85 W4(1,1,...,1, sn)dy(1, 1,..., 1, 82) 
pF = ? ? ? ? Dv 3 > > 3 60 
ae) ESE: Olli ls ces lyse) a 


From the definition of ¢,, d)(1,1,...,1,s,) = Es?*" [see above (52)]. 
Hence from (50) and applying (138), (14) we have 


1 
O} (se) = 7 W(L, 1,» Se) be) (61) 
where ¢ = limnto Cn and Es, = $-(s,). Hence whenever pu > 0, 
1 
of Pe (-) 
Le 


is a generating function. Now, it can be easily verified that corresponding 
to each j the unique solution 69 (so, ...,Sk) = 0;(S0, $1, ..., Sz) satisfies 
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an equation similar to (55): 


k 
0;(So, $1,..-, SR) = | 65(1, 50,82, ++) Sk—2, Sk—-18k) [J] sz 
i=0 


R 
of (1 _ IL 7") Yj (s0,81, o ++ y Sp-Q, 5.18) | 
l= 


x dv (So, S1,--+,5 Sp) (62) 


The family of such solutions 6; are linearly independent. If the generating 
function of P; (the equilibrium distribution of S), (so, s1,..., Sz), is 
such that 


ko 
$(0, s1,.-., 8p) = ud’ cj J] st! (63) 
j i=1 
where the sum on the right is over all indices j = (0, j1,... , Jz) which are 
in A, then ¢(So, s1,..., 5) has the unique representation 
(So, S1,-- +, Sk) = Y,’ C59; (So, $1, --- , Sr) (64) 


° 


J 


Notice that corresponding to each j there exists a sequence P;(j), not 
necessarily nonnegative, such that 


dX Pili)si? sit... si = 0;(s0, $1,.-., Sx) (65) 


Hence P; itself has the representation 
Pi= Y c;PiQ) (66) 
j 


Furthermore 0% (sx) from (61) corresponds to a nonnegative summable 
sequence. From (59), starting with r = k — 1 and going backwards tor 
= 0, we can show that for each j, P;(j) is the convolution of absolutely 
summable sequences and hence 


> |Pili)| < © (67) 
From eqs. (61), (65), and (66), when 3c; = 1, 


DX Pi = DY’ cjOF (1) 


icA j 
=>'cj=1 (68) 
We will now show that there is a finite number of linear equations 
derived by substitution of (66) into (49) which uniquely determine {c;} 


and hence P;. Let us denote the elements of the transition probability 
matrix of S, Pr {yn+i = ilyn = jj, by Ty. Then from (40) 
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Pi= Dd TyP; (69) 
jeA 
Substitution of (66) yields 


De’ CmPi(m) = YT ij 0’ ¢mPj(m) (70) 


je m 
which is a set of linear equations for cm. Hence any solution {d,,} of (70) 
has the property that Q; = 2,, dmP;(m) satisfies (69), hence 


Qi = Y Ti; (71) 
je 


Now let Tf be the n-step transition matrix of S. Then using (71) we 
have . 


Qi= DU THQ; (72) 
jeA 
Since S is positive recurrent 
lim T} = Pi (73) 
nto 
Furthermore since 2;|P;(j)| < ~ for each j, 2;|Qi| < 0. Hence taking 


limits of both sides of (72) and interchanging limit and sum on the right 
hand side of (72) we have 


Qi = lim E73 Q; 


nto j 
= Pid Qj (74) 
j 
However, if 2; d; = 1 then, from (68), 2; Q; = 1. Therefore 
Qi = Pi (75) 


Since P;(j) are linearly independent, c; = d, for each j, and {c;} are the 
unique solution of (70). 


Remark 1: A similar set of equations can be obtained by substituting 
(64) into (55) and equating the coefficients of like powers on both sides 
of (55). 


Remark 2: Note that (0, s1,...,$,) = win the case when, for each 1, 
J, a i > 0. Hence (58) may be used repeatedly to obtain an expression for 
(So, $1, -.-, Sk). Herbert® considered this model when ¢ = 1. 

In the alternate formulation (0, u;,..., uz) is a multinomial. 
Moreover, from (34) and (35), since aj > 0,i =1,...,¢, Yon =O x)_, 
- =0,1=1,..., ¢, which implies yz, = yp—1n. Hence (0, wi, ..., uz) is 
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independent of uz. From (58’) and (58”), ®(s, uw, .. oe may be ex- 
pressed in terms of ®(0,s,...,S,U4,.. » Up—j- vJjJ= .,k —2,and 
£(0,s,...,8). If we lets > 0 in this EapressiOn, and mee ®(0,u1,..., 
ux) with the finite part, we obtain a system of homogeneous linear 
equations for the coefficients in the multinomial. In general, we also 
obtain a (consistent) set of homogeneous linear equations from finiteness 
conditions. 


Vil. GENERATING FUNCTIONS FOR MARGINALS AND FINITE 
SOLVABILITY 


The joint distributions of (yo, y1,..., vz) have (k + 1) arguments. We 
will see that we can reduce the problem to “k + 1 one-dimensional 
problems” when we are only interested in the marginal distributions of 
Yo. ¥1,--+»¥R- Let us denote the generating functions of y; by ®;(s) and 
those of Urn by ¢;y(s). Then 


l 
gifs) =¢(1,1,...,8,1,...,1) 
i+1 k-i 
= @(s,...,8,1,...,1,1=0,...,R (76) 
From (55) we then obtain forr =0,...,k —1 


+1 
PaO | seats) Seg aie, eet 1) Jon(s) 
and 


on (s) = [soz (s) + (1 — s—G(0,1,...,1, 5) ]beo(s) (77) 
Note that 


r 
OLOr ied 8S jc 1D 


r k-r 
=O: (0.6.0.0 3S) lesa) = Tench 


Therefore once the cj; have been determined from the method pre- 
sented above, Eq. (77) gives the marginal distributions. Once again we 
can translate (77) into linear equations for the distributions themselves 
as in (15). The marginals are finitely solvable in the sense that a finite 
number of components of the marginal distributions can be solved for 
from a finite number of linear equations. 

For eachr = 1,2,...,k let y,; be the coefficient of s/ in the polyno- 
mial 


r 
PO, Lowey 84 oa) 
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denoted by c,(s). Equating coefficients of like powers of s/ on both sides 
of (63) after setting s; = 1 fori # r yields 
Vj =e DL Cj (78) 
Jr=] 
Therefore since the c;’s can be determined as the solutions to a finite 


system of linear equations, so can the y,,;’s. 
Let 


bea ; j 
or(s) = > TjS!, i,j => Wrj 
j=0 i=0 


and F,(s) = Zj2o IL,s/ for |s| < l andr =0,1,...,k. Then F,(s) = 
o,(s)/(1 — s), and eqs. (77) become 


F,(s) = s—"[F,41(s) — ¢r41(s) Orv (s),7 = 0,1,...,kR -1 (79) 
_ bry (s)cx(s) 
re bkv(s) — s oY 


From (79) we can show that, for eachr = 0,1,...,k — 1, {IL eo are 
determined from {I1,+1,;}“6’*! by a finite set of linear equations, for any 
N. Let the sequence {6,;} correspond to s~}[F,+1(s) — ¢,+1(s)]. From the 
definition of F+1(s) and c,+1(s) it follows that I,-+1,0 = Yr+1,0. Therefore 
6,; = 0 for j <0 and 
6rj = U-+1j+1 — Yr+1j+1 for 1 Sj + 1 < degree of c,+1(s) 
= Il-+1,+1 for j + 1 > degree of c,+1(s) (81) 


From (79), the sequence {II,;}"4" is the convolution of {6,;} with {p,;}—the 
sequence of probabilities corresponding to the characteristic function 
rv (s). Therefore, since the sequence {p,;} is known a priori, we can find 
II; as: 


J 
Tj = 20 9. j-Pri, J = 0,1,...,N4r (82) 
i=0 


Hence, we observe that {IIo;}) can be determined as solutions to a finite 
system of linear equations using {II;, ay 

In order to find {II; sg we proceed as in (15). Equating the coeffi- 
cients of like powers of s/ in 


Prv(s) 


. Tl; gi = (83) 
Pe Gs 
yields: 
Iho =1 
hj = (Pai so) es fee 3 prillj,j-i) fon 
To Apes pee ees N+R (84) 
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Therefore {I1},;}5* can be determined uniquely as solutions of (84). From 
(80) and (83), {II eo. is the convolution of {II;,;} with {z;}, 


J , : 
May = 2 Wh jive = 01... N+R (85) 
l= 


Therefore we have shown that each of the II,;, and hence the marginal 
distributions z,;,r = 0,1,...,k, 7 =0,1,...,N +7, can be found, for any 
finite N, as solutions to a finite system of linear equations. 


Vill. A LIMITING CASE 


For each m let dj, be a nondecreasing sequence of nonnegative inte- 
gers such that 


(i) djo = 0, dj. =j;j7 =0,1,...,R 
mf o 


(ii) am div — dj-1,m = ©, J = if Lica | k (86) 


We define a sequence of processes {z”"} which will be time-scaled versions 
of z,. Let 


mo 
en = 
l 


ts 
ihe 


i yt 
a; Xn-djm 


We observe that z} is the same as zp, and z® is the “fastest” version of 
Zn, in the sense that all the packets triggered by x!, are bunched together 
and arrive at the same time. As m increases the different delayed con- 
tributions of x!, are spread farther and farther apart in time. The limiting 
case can then be interpreted as the ‘“‘slowest”’; see Ref. 7. 

Let {n,}, n = 0, 1, 2,..., be a sequence of independent identically 
distributed random variables such that for each n the distribution of 7, 
is the same as that of z,. We will show that 7, then corresponds to the 
slowest case: the finite dimensional distributions of the processes {z7"}, 
m =0,1,... converge to the corresponding distributions of {n,}.as m + 
o. Indeed, let ny < ng <+» <n, be nonnegative integers. Then 


s 
Pr (2, = ty Zing = be,...,2n, = be} = I] Pr {2m = i; (87) 
i= 


for large enough m, in particular for every m such that djm — dj-1,m > 
ns —n1,j =1,...,k. However from the definition of n,, Pr lZn, =ij= 
Pr {nj = 1;}. Therefore from the independence of n,, and (87) 

PriZt S12 A 103 sa Sa tg 


=P tng = 11, tng = Lo, oe ey Ins ig} (88) 


for large enough m. 


1766 THE BELL SYSTEM TECHNICAL JOURNAL, NOVEMBER 1977 


We now define a sequence of processes b7’, b> corresponding to 2”, 
Nn respectively. Formally let 


ba = (bP ae Be rahe 
bri = = (br _ 1)* sa FS (89) 


Since Ez7’ = Enp, if Ez, <1 then for each m, limy}~ b7 = b”™ is a well- 
defined random variable, and so is b® = lim,}. bn. We can then show 
that 


lim b™ = b@ (90) 
mito 


from Theorem 22 in Ref. 9, since z™, n, are nonnegative. Hence the 
distribution of b” approximates the distribution of b™ for sufficiently 
large m. Therefore for each J 


lim Pr {b™ <j} = Pr {b? <j} (91) 


mio 


Therefore b® is the steady-state queue size corresponding to the 
“slowest” version of z,. Let 


bx (81,.--58¢) = E (11 st) 


Then it is easy to verify that Es: and Es" are given by 


k 
p= oz (suk, suk, ..., s#h) and 62 = TI oy (s%, s%,..., sf) 
1=0 


respectively. If ¢° = Es®° and ¢° = Es®” then 


ye s7)o2(s)u 


1 —s—!42(s) 
A= 8-426) 
OT = 519268) ee 


In the special case when ¢ = 1, and (omitting the superscript) a; = 0 
or 1 for each J, we have an interesting special relationship between ¢° 
and $°%. Let f? = Pr {b° < n}, fe = Pr{b° <njand Fo = dfes", Fe = 
Lfrs”. Then F° and F@ are ¢°/1 —s and $°/1 — s respectively for |s| < 
1. We will show that 

in = i (93) 


equivalently 
Pr {b? <n} = Pr {b° < ny,z} (94) 
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Let w bea a upth root of unity. Then for |s| <1 
= Fw L s)= = 1 u 


Uk i=0 Uk i=0 $x (s#*) — ws 


_ uldx (s#r)]He 


~ [oe (out)oe — gee OY 


Therefore 


=> foie Sork 
n=0 


(95) 


(96) 


Since f° and f? are both increasing and bounded by 1, (96) shows that 


(93) holds. 
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Techniques for reliably estimating the power spectral density 
function for both small and large samples of a stationary stochastic 
process are described. These techniques have been particularly suc- 
cessful in cases where the range of the spectrum is large. The methods 
are resistant to a moderate amount of contaminated or erroneous data 
and are well suited for use with auxiliary tests for stationarity and 
normality. Part I is concerned with background and theoretical con- 
siderations while examples from the development and analysis of the 
WT4 waveguide medium will be discussed in Part II, next issue. 


|. INTRODUCTION 


The problem of estimating the spectrum of a stationary time series 
has appeared frequently in the scientific literature and myriad ap- 
proaches have been suggested. Nonetheless it became apparent during 
the course of the development of the WT4 waveguide system that these 
methods were inadequate for many of the data sets of interest. The 
techniques presented here were therefore developed. 

It is commonly stated that the method selected to estimate a spectrum 
depends on the ultimate use of the estimate, and unfortunately to some 
extent this is true. The method described below is felt to represent an 
advance in that the basic technique works well in a variety of cases which 
previously would have required individual treatment. The loss calcu- 
lations reported in Anderson et al.! are indicative of its accuracy. 

The procedure which has evolved for estimating spectra can best be 
described as robust adaptive prewhitening. Such methods have three 
distinct stages: formation of a pilot spectrum estimate, using this esti- 
mate to design a prewhitening filter, and finally giving the result as the 
ratio of the spectrum of the filtered data to the power transfer function 
of the filter. This method is potentially both efficient and robust. The 
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efficiency of a statistical estimation procedure is the fraction of the in- 
formation, in the sense of Fisher,? conveyed by the estimate about the 
parameter being estimated to the total information on this parameter 
inherent in the data. An estimation procedure is robust if it remains 
efficient over a wide range of conditions and is relatively immune to a 
small fraction of outlying or erroneous data. 

For the sequential method described here to be efficient, the pilot 
estimate must be designed to have a large dynamic range at the expense 
of frequency resolution. The second spectrum estimate, which works on 
the filtered data, uses the opposite choice and so is chosen on the basis 
of frequency resolution. This can be done without incurring a large 
penalty in loss of effective dynamic range as this information, acquired 
by the pilot estimate, has been transferred to the filter specification. In 
one meaning of the term this procedure is robust in that it can normally 
handle situations where either estimate alone would fail. By using a 
nonlinear filter for the prewhitening operation the procedure may also 
be made robust in the sense that it is resistant to moderate amounts of 
erroneous or contaminated data. 

In this method the pilot estimate of spectra is a combination of several 
direct estimates of spectra computed on subsets of the data using a 
window defined by a prolate spheroidal wave function. Using this esti- 
mate as a basis an autoregressive model of the process is formed. This 
model is then used to generate a nonlinear prediction error filter. The 
output of this filter consists of prediction residuals from a modified data 
sequence and is quite immune to occasional isolated errors in the 
data. 

Section II gives an overview of the complete estimation procedure so 
that the descriptions of the individual stages of the process are taken 
in the proper perspective. Section III is a review of properties of direct 
estimates of spectra which are used for both the pilot and final estimation 
procedures. Sections IV to VIII describe the several stages of the esti- 
mation procedure in detail. While these sections contain some examples 
they are primarily concerned with theory and background. Part II will 
consist primarily of examples and comparisons with standard tech- 
niques. 

It should be emphasized that the same approach is used for both short 
and long data sets and that the only difference between these cases is 
one of detail and not philosophy. We define a short time series as one 
which cannot be subdivided into subsets having almost uncorrelated 
spectrum estimates. 

Since this technique is basically nonparametric, it is frequently asked 
whether a parameterized estimate of spectrum might not give better 
results. It has been shown by Arato® that only for the autoregressive case 
can a process be described by a fixed number of sufficient statistics and 
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that in general the number of sufficient statistics increases with the 
sample size. As a result, efficient parametric estimates are not likely to 
be even conceptually simpler than the nonparametric estimates used 
here. 

It is also asked why maximum-likelihood techniques are not used di- 
rectly, and, while asymptotic results on parametric maximum likelihood 
estimates of spectra are available in Whittle,* constructive procedures 
for obtaining nonparametric maximum-likelihood estimates of the 
spectrum of a stationary Gaussian time series are unknown. It is, how- 
ever, possible to check if a given estimate is maximum likelihood or not. 
This test, described in Thomson,° depends on the Karhunen-Loéve 
expansion of a random process (see Loéve®). In this test the data is ex- 
panded in terms of the sample eigenfunctions of the spectrum estimate, 
and, if this estimate is maximum likelihood, the expansion coefficients, 
Gp, will satisfy the conditions 4? = \, in which 4,, are the corresponding 
sample eigenvalues. By the Szeg6 theorem (see Grenander and Szeg6") 
this comparison is asymptotically equivalent to comparisons on the 
spectrum at a frequency spacing of 1/T. This agrees with the conven- 
tional Rayleigh resolution and heuristically a spectrum estimate with 
this resolution and low bias is likely to be efficient. This argument pro- 
vides the motivation for the present technique. Simple data windows 
with frequency resolution close to 1/T do not provide enough bias pro- 
tection. Moreover this is not just a result of not having chosen the right 
“simple” data window but the result of fundamental characteristics of 
the Fourier transform (see Landau and Pollak’). Data windows like the 
4a prolate spheroidal wave function which provide the protection from 
bias have frequency resolution on the order of 4/T and so are inefficient 
from this viewpoint. It must be emphasized that the sequential approach 
used here potentially has both limitations since it cannot resolve details 
spaced by 1/T in frequency when their levels are more than 4 or 5 decades 
apart. On the other hand if the spectrum is not quite so pathological and 
varies “slowly” over 10 to 15 decades then the method can provide fre- 
quency resolutions approaching 1/T with relatively low bias. 


ll. SUMMARY OF THE ESTIMATION PROCEDURE 


2.1 Data preparation 


At the beginning the data is plotted, and serious outliers, missing 
values, etc., edited by use of either interpolation or successive prediction 
and interpolation. These predictors and interpolators are the optimum 
linear forms based on previous spectra estimates of a similar process or 
on assumed valid data from the current sample. It is also frequently 
necessary to remove the mean value function of the “cleaned” data. This 
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is always done in the analysis of individual tubes to eliminate the cur- 
vature resulting from gravitational sag. 


2.2 Pilot spectrum estimate 


For the remainder of this paper we assume that the available data is 
a sequence of samples {x;}, t = 0, 1--L, and that the sampling interval 
has been normalized to 1. Consequently the normalized Nyquist fre- 
quency is 4. Both because the notation is more compact and also as a 
reminder that the basic processes are continuous! most operations will 
be denoted by integrals. In the actual computations most of these inte- 
grals are replaced by simple sums but on occasion spline approximations 
to the integrals (see Aronson!") are used. The frequency variable will be 
denoted by f with w = 2zf. 

The initial spectrum estimate is normally computed using a variation 
of Welch’s!! method: the basic data set is divided into k overlapping 
subsets each of length T and offset from the previous one by a distance 
b. The data from each subset is tapered using a zero order prolate 
spheroidal wave function, with parameter c = 47 and the Fourier 
transform of the result computed. The raw estimate of spectra on the 
jth subset, S;(w), is then the squared magnitude of the transform so that 
its univariate distribution is proportional to a x2. The use of the prolate 
data window guarantees, under simple conditions, that the bias of the 
estimate within each subset is of purely local origin and that estimates 
separated by more than 2c/T in frequency are essentially uncorrelated. 
However, to account for the correlation induced by the tapering the total 
number of degrees of freedom must be reduced. These effects and the 
bivariate distribution of the estimates is discussed in Section 3.2. 

Because the raw estimates, S;(w), are very volatile it is often desirable 
to smooth the different subset estimates. These estimates, smoothed 
to have v degrees of freedom, will be denoted by S; (w). In the original 
Welch technique the pilot estimate of spectrum, S(w), is the arithmetic 
average of the subset estimates. When the data contains outliers it is 
advantageous to replace the simple average with a robust combination 
of the subset estimates as discussed in Section V. Both because it is based 
on subsets of the data and because of the heavy tapering implied by the 
use of the parameter c = 47 (see Section II]) the pilot spectrum estimate 
has poor frequency resolution compared to the final estimate of spectra. 
For reasons discussed below excessive resolution in the pilot estimate 
is frequently counterproductive and this technique produces a stable 
estimate with adequate bias protection in situations where the range of 
the spectrum is very large. 


1 The paper by Dzhaparidze and Yaglom® contains information on the complexities 
induced by sampling basically continuous records. 
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2.3 Tests for stationarity 


Large data sets can be tested for stationarity using the method de- 
scribed in Thomson.!? Briefly the approach compares the different 
subset estimates, S; (w), using Bartlett’s M statistic for heteroscedast- 
icity of variance between subsets at constant frequency. Equally spaced 
samples of the test statistic, M(w;), are then pooled and tested for con- 
formance to the distribution expected for homogeneous samples. 


2.4 Construction of autoregressive models 


Stationary time series have four generally accepted canonical repre- 
sentations; Cramér’s orthogonal increment spectral representation, the 
Karhunen-Loéve expansion, the moving average, and autoregressive 
models. Of these the autoregressive model is perhaps the most useful 
for making inferences on the structure of the process. For further in- 
formation see the review paper by Kailath.!8 

Most autoregressive methods either begin with a sample autocorre- 
lation function and solve the Yule-Walker equations directly (Ma- 
khoul!*) or else resort to a variation of Wiener spectral factorization 
(Whittle!>) applied to an estimate of spectra; neither approach is entirely 
satisfactory. For the estimation of waveguide spectra both methods have 
been used and in Section VI a method combining the better features of 
both is discussed. In cases where the range! of the spectrum is relatively 
small, solving of the Yule-Walker equations using Durbin’s modification 
of the Levinson algorithm (see Section VJ) is satisfactory. In this case 
the autocorrelations used are obtained by Fourier-transforming the pilot 
estimate of spectra. When the range of the spectrum is larger the Wiener 
technique is more stable but results in a very long predictor. Backward 
application of the Levinson algorithm may then be used to generate a 
more compact representation. In both cases the order p of the autoreg- 
ressive representation has usually been chosen on the basis of Parzen’s!6 
stopping rule and the innovations variance corresponding to the pilot 
spectrum S(w). Details of the procedure are given in Section VI. 

The autoregressive representation has an intuitive explanation in 
waveguide applications in which the prediction can be thought of as 
_ analogous to a local “warped normal mode” representation and the in- 
novations process the changes required in the field configuration to 
maintain the “local” character. The casual nature of the autoregressive 
representation corresponds to propagation in the forward direction so 
that the field configuration at a given point reflects distortions which 
have been passed but not those in the future. 


*The range of a spectrum refers to the logarithmic range or the ratio 
max {S}/min {S}. 
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2.5 Prewhitening, robust filtering 


The autoregressive model formulation gives the casual filter which, 
for fixed impulse duration, p, has minimum output power. The residual 
sequence or output of such a filter (known as a prediction error filter) 
is the difference between the observed and predicted values of a data 
sequence using the previous p data points as a base for the prediction. 
When the autoregressive model is correct the residual sequence will be 
serially uncorrelated and have a white spectrum. When the data contains 
outliers the effect of such filtering is to contaminate the p residuals 
following each erroneous point. 

The robust filter algorithm is a nonlinear procedure based on an au- 
toregressive model which is designed to reduce the effects of occasional 
outliers. The output of this filter or the modified data sequence is an 
estimate of the uncontaminated process. This sequence is formed by 
comparing successive input data points with the value predicted from 
the modified sequence. In regions where the prediction errors are “small” 
relative to the innovations variance, the modified sequence is essentially 
a copy of the input data. When the prediction errors are “large,” the 
corresponding points of the modified data sequence are the predictions 
rather than the data and for intermediate prediction errors the behavior 
depends on a weight function. When the modified data sequence is used 
as a basis for the final estimate of spectra, the prediction error sequence 
is the difference between the predictions and the value of the modified 
data sequence. For uncontaminated data this corresponds to the output 
of the linear prediction error filter but when a large error is present the 
algorithm has two effects: first, the large output residual is replaced by 
a zero; second, because of the feedback nature of the method, propaga- 
tion of the error into subsequent predictions is greatly reduced. As with 
all methods which alter or ignore extreme observations a compromise 
must be drawn between rejecting some valid data and accepting occa- 
sional errors and, in the robust filter algorithm, this compromise is re- 
flected in the choice of weight function. In Section VII a weight function 
motivated by the normal extreme value distribution which has both 
intuitive appeal and desirable mathematical properties is described. 


2.6 Final estimate of spectrum 


The prediction residuals, or output of the prediction error filter, are 
the sequence which has minimum power for a filter whose impulse re- 
sponse has duration p. Consequently in the frequency domain the effect 
of such an operation is necessarily to reduce the highest parts of the 
spectrum first. As the complexity of the filter is increased the residual 
spectrum approaches a constant at which point further improvement 
is impossible. In practive finite order autoregressive filters seldom attain 
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this limit but rather have the effect of reducing the range of the spectrum, 
usually without following any fine structure which is present and, as a 
result, information describing the fine structure is left in the residual 
process. On occasions when the autoregressive fit is forced to follow too 
fine structure in the spectrum the spectrum of the residuals may be lo- 
cally more complex than the spectrum of the original process.* 

Since the range of the spectrum has been reduced the procedure used 
to estimate the spectrum of the residuals is designed to have high-fre- 
quency resolution at the expense of sidelobe suppression. 

When the nonlinear version of the prediction error filter is used it is 
commonly observed that the pilot spectrum, estimated from the con- 
taminated data, is considerably higher than the final estimated spectrum 
at frequencies where the spectrum is small. So that these differences are 
not obscured with bias the pilot final taper must be such that the cor- 
responding spectral window decays significantly with frequency and 
consequently tapers such as the Taylor equiripple design (see Rife and 
Vincent!”) are inadvisable. The window which has been used most for 
this purpose is Tukey’s spliced cosine taper. For long data sets this 
window is satisfactory but with very short sets, for example individual 
waveguide tubes, the first sidelobe of this window is too high and a more 
complex window described by a series expansion in prolate spheroidal 
wave functions is used. 

The final estimate of spectrum is based on an approximation intro- 
duced in Grenander and Rosenblatt,!® which is that the predictor and 
prediction residuals are statistically independent. Under this assumption 
the final estimate of spectrum will be the spectrum of the residuals di- 
vided by the power transfer function of the prediction error filter. 


2.7 Smoothing 


One of the most commonly recommended operations in spectrum 
estimation is that of smoothing the raw estimates by means of local av- 
eraging over frequency. Contrary to these recommendations the final 
estimates of spectra are almost never smoothed. Moreover, in cases where 
“smoothed” estimates of spectra are used, the smoothing is frequently 
the result of nonlinear and adaptive procedures. Such smoothing is useful 
in plotting applications, and for improving the stability of pilot spectrum 
estimates from short data sets. Certain nonlinear smoothers are also very 
useful for finding low level lines in complex spectra. 

The general philosophy of these methods has been to test the raw 
spectrum for local homogeneity: when the local spectrum appears to be 


+ For spectrum estimation problems a good measure of complexity is 


2380/9 
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homogeneous it is smoothed, but in cases where the raw spectrum ex- 
hibits variations greater than normal, a typical response is to reduce the 
width of the smoother. A second approach which is used is to initially 
“smooth” the raw spectrum using a robust nonsymmetric location es- 
timate and then to put the peaks back on the basis of “inverse influ- 
ence.” 


ill. DIRECT ESTIMATES OF SPECTRA 


In both the pilot and final phases, the spectrum is estimated by the 
so-called direct method and while the parameters and application of the 
estimator are different in the two cases, the basic form is the same. In- 
formation on direct estimates is available from several sources, for 
example Blackman and Tukey,!9 Jones,2922 Tukey,2? Koopmans,”4 
Brillinger.25 In this section properties of the direct estimate are reviewed 
and compared to the indirect estimate; the role of prolate spheroidal 
wave functions as a means of reducing the bias of the estimate is de- 
scribed and compared to standard data windows. The next subsection 
describes the variance of the estimates with emphasis on characteristics 
of prolate windows and smoothing when the estimates included in the 
smoother are correlated. The final subsection is concerned with Welch 
estimates and a technique for choosing the optimum subset spacing. 

The direct estimate of spectrum is defined by 


Sie f.  eietD(t)x(t)dt| (1) 





In this definition the data, x, is defined on the domain [0, 7], w is radian 
frequency, and D is a data window or taper. The data window is nor- 
malized according to the convention 


T 
f, D%(t)dt = 1 (2) 


so that the resulting spectrum is interpretable in physical units. 
Almost all of the published estimates of spectra are either direct es- 
timates, smoothed direct estimates, or rational fits to direct estimates. 
When D is constant Sp is the periodogram. Smoothing the extended 
periodogram? with appropriate weights corresponds to the various in- 
direct estimates. Similarly an autoregressive or “maximum entropy” 
estimate may be regarded as an all-pole rational fit to the extended 
periodogram and Pisarenko”¢ estimates constitute a generalization of 


t In the simple periodogram estimates are computed at a frequency spacing of 1/T and 
the corresponding autocorrelations are circularly defined. A frequency spacing <1/2T 
is used in the extended periodogram and its Fourier transform yields the common auto- 
correlations. 
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this idea.? The notable exceptions are the Whittaker periodogram and 
the Burg estimate (see Section 6.4). 

The application of smoothers or curve-fitting procedures to the basic 
estimate conceals its true nature and the fact that the properties of these 
estimates are controlled primarily by the data window D. For example 
it is commonly stated that the fundamental uncertainty in spectrum 
estimation is between resolution and variance and more papers than it 
is convenient to list have worked on better “lag windows” to minimize 
this conflict. Unfortunately the emphasis on this secondary problem has 
masked the primary uncertainty between resolution and bias. The basic 
problem with indirect estimates and the lag window approach is that it © 
represents an attempt to patch the periodogram. A more logical approach 
is to start with a better basic spectrum estimate. 

Despite its simplicity the direct estimate is not well understood. In 
particular the differences between direct estimates using data windows 
and indirect estimates using lag windows are frequently confused. 

The expected value of the direct estimate (1) may be written 


é T pT. 
E{Sp(w)} = f. f. eielt-wWD(t)D(u)F{x(t)x(u)} dt du (3) 
0 
For second order or covariance stationary processes the autocovariance 
function is defined by 
R(r) = Efx(t)x(t + 7)} (4) 
and may be represented in terms of the spectral density function by using 


the Wiener-Khintchine relation 


1 
R(x) =5- fe S(w) dw (5) 


and denoting the Fourier transform of the data window D by D one ob- 
tains 


E{Sp(w)} = S(w) *|D(w)|2 (6) 


where S is the true spectrum of the process and * indicates convolution. 
Since D is a time-limited function D is an entire function of w so that 
the direct estimate is biased for all spectra which are not white. The 
function | D(w) |2 is known as the spectral window of the estimate. 

An alternative description results from expressing eq. (3) in terms of 
the autocovariance function, R, of the process as 


E{Sp(w)} = fe Lo) RO dt (7) 


? The Capon?’ estimate, while superficially similar, is intended for estimating the 
magnitude of periodic components in a background of a known covariance structure. 
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where the convolution D*D has been identified as an “equivalent” lag 
window, Lp(r). Because of this identification characteristics of the in- 
direct estimate of spectra, 


P / ae ‘ 
Si(w) = fea Buln) dr (8) 
using an unbiased estimate of autocovariance 


A 1 T-|r| 
Rus) or x(t)x(t + |r|) dt (9) 


are often used incorrectly to describe the direct estimate, S p(w). Except 
for their first moments these two estimates have few properties in 
common: one very important difference is that the direct estimate is 
positive while the “equivalent” indirect form need not be. Also, because 
their common spectral window enters the estimate in fundamentally 
different ways, the variances of the two estimates are different. 


3.1 Minimum bias estimates and prolate spheroidal wave functions 


The most convenient description of bias induced by the data window 
is through the spectral window |D|? as expressed in eq. (6). The effect 
of this convolution is to change the apparent distribution of power in a 
complex manner and, since all windows cause some redistribution, a 
minimal requirement is that the indicated power be left “close” to its 
original location. Defining “close” to be within a tolerance () of w we re- 
quire that the broadband bias, i.e., bias from outside (w — Q, w + 2), be 
small. Denoting this bias by Bg(w) and the integral over frequency with 
the section (w — Q, w + 2) excluded by # we have 


Ba(w) =5- Ff Sw- DIPS (10) 


From the definition of a direct estimate and the convolution theorem 
the broadband bias in a particular sample is 


Baw) = | FE(w — ND(S) ds |? (11) 
where &(w) is the spectral representation of x (see Doob,?® chapter 10). 
By the Cauchy inequality this bias may be bounded so that 
A 1 1 as 
Bao) <— F |#@- OP do>--F [Dw oP ds (12) 
24 24 


The first factor of this inequality depends only on the process and, as 
the integrand is positive, is simply bounded by adding the integral from 
w — Q tow + Q and identifying the result using Parseval’s theorem. The 
second factor in the inequality depends only on the data window, D, and 
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expresses the energy in D outside —Q, Q. D is a time-limited function of 
unit energy and this inequality is minimized when D is a prolate sphe- 
roidal wave function. The fundamental role of these functions in relation 
to Fourier transforms and related problems have been described in a 
remarkable series of papers by Slepian and Pollak,29 Landau and Pol- 
lak,®.30 and Slepian.?! When the bounds for both integrals are combined 
the result is that 


Bp(w) < 62T(1 — doo(c)) (13) 


where 6? is the sample variance, c = 2T/2, and doo(c) is the largest ei- 
genvalue of the integral equation 


1 sinc(t —s 

Mable) = f(s) ds (14) 
-1 a(t—s) 

Tables of the eigenvalues of this equation have been published in Slepian 

and Sonnenblick®? and asymptotic descriptions given by Slepian.?? From 

the latter reference 


1 — doole) © 4V ce ~¢ (15) 


As the width of the guard band, Q, increases this bound decreases rapidly. 
For exploratory time series analysis and the formation of pilot spectrum 
estimates a very convenient value of c is 4a for which 1 — App © 3 X 10719. 
In Thomson e¢ al.34 empirical studies show that direct estimates using 
this window are generally superior to several other spectral estimates 
in common use. Other examples are contained in Thomson.° Windows 
using approximations to prolate spheroidal wave functions have been 
described by Kaiser,®° Eberhard,®6 and in fact the Parzen?” window can 
be considered as a fourth-order successive approximation to the 47 
prolate window. 

Figure 1 shows the 47 prolate data window (and several other windows 
described below) and the low weighting given near the ends of the data 
are evident. The corresponding spectral windows are shown in Fig. 2, 
and here the reason for using the 47 prolate taper in situations where 
the spectrum varies over large ranges is most evident. The frequency 
scale of this plot has been normalized to units of 1/T so that by a fre- 
quency of 4/T the spectral window corresponding to the 47 taper has 
decayed by more than 10 decades. It should be noted that the curves for 
the other windows represent envelopes of the spectral windows. The 
actual spectral windows are similar to that shown for the compound 
prolate window and decay in an oscillatory manner. 

When the range of the spectrum is known to be small it is clear that 
the use of this window is inefficient in that the frequency resolution is 
much less than it is for windows with higher sidelobes, and several al- 
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Fig. 1—Comparison of data windows. 


ternatives are available: no tapering, ad-hoc tapers, and prolate tapers 
with lower values of c. 

Very few spectra resulting from physical processes are so uninteresting 
that the “elimination” of tapering is ever advisable; in this case the taper 
actually used is 1/ T over (0, T) and 0 elsewhere. This “default taper” 
has 

T (= wT /2 ) 2 
wT'/2 


as a spectral window so that, as shown in Fig. 2, the first sidelobe is only 
~13 dB down from the central maxima. 

Of the various ad-hoc techniques, Tukey’s?? spliced cosine taper is 
perhaps the most useful and it has been used for many of the final esti- 
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Fig. 2—Envelope of spectral windows. 


mates of spectra used in the WT4 project. Since this taper, shown in Fig. 
1, weights the data in a much more uniform manner the corresponding 
spectral window, Fig. 2, has a narrower center lobe than the 47 window 
and the sidelobes decay much faster than those of the default win- 
dow. 

Unfortunately the first few sidelobes of this window are too high for 
it to be usable in many applications where accurate estimates of the fine 
structure of a spectrum are required. This leads to considering the 
spheroidal wave functions again and for maximum concentration in a 
bandwidth 1/T the appropriate value of the parameter c is 7. As before 
the maximum concentration is achieved by using the function of order 
0 but for the present application a better compromise can be obtained 
by using a linear combination of the functions of order 0 and 2 with the 
coefficients determined by the additional constraint imposed by re- 
quiring that the first two sidelobes be minimized. This “compound 
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prolate” taper is also plotted in Fig. 1 and it is clear that the weight is 
less extreme than the 47 taper but distinctly different from the spliced 
cosine form. From the plot of the spectral windows it can be seen that 
the main lobe of the compound window is almost as narrow as that of the 
spliced cosine and also that the first sidelobes are down 27 dB instead 
of 13 dB. Since the widths of the main lobes are all very close to the same 
width, this gain in performance is essentially free and results from the 
superior characteristics of the prolate functions. It might also be men- 
tioned that the usual objection to the use of the prolate spheroidal wave 
functions, namely that they are “impossible” to compute, is false and 
that by using Horner’s rule together with the expansion given in Flam- 
mer,°8 Section 3.2, they may be computed very rapidly. Appendix A gives 
expansion formulae for the z and 47 prolate data windows. 

In anticipation of Section VI it is also interesting to compare the bias 
of the estimates of autocorrelation obtained by transforming the various 
spectrum estimates. From eq. (7) it is apparent that such estimates of 
the autocorrelation function at lag 7 will be biased by the factor Lp(7). 
These lag windows are plotted in Fig. 3. From this figure it can be seen 
that the bias imposed on the low-order autocorrelations by the win- 
dowing techniques is much less than that resulting from the common 
positive definite estimate [obtained by replacing the factor T — |r] in 
eq. (9) with T] corresponding to the simple extended periodogram. It 
should be noted that if this factor is divided out the resulting unbiased 
estimate is not positive definite and frequently results in negative 
“prediction variances.” For fitting autoregressive models, the low-order 
autocorrelations are crucial and, as can be seen from the insert in Fig. 
3, for 7/T < 0.01 the bias obtained using the 47 prolate window is lower 
than that obtained from the extended periodogram on data sets 10 times 
as long. The scale of such comparisons can be best judged by noting that 
the one-step autocorrelation in the field evaluation test curvature data 
is about 0.99983. 


3.2 The distribution of direct spectrum estimates: liftering' 


The preceding sections were adressed primarily to the problem of bias 
in direct spectrum estimates without particular attention being paid to 
their variances or distributions. Since reliable interpretation of spectrum 
estimates requires understanding of both their distributions and the 
correlations between estimates, the following sections treat these and 
the closely related problem of smoothing. Because of the correlations 
induced by the data window, the variance of smoothed direct estimates 
depends both on the smoothing weights and on the data window. 

As mentioned in the introduction, the final estimate is rarely 


t See Bogert, Healy, and Tukey”? for definitions of these terms. 
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Fig. 3—Comparison of lag windows. 


smoothed, but in the formation of the pilot estimate smoothing is a 
critical step. The primary reason for smoothing the pilot spectrum es- 
timate is to obtain a more accurate autoregressive model. Because the 
direct estimate is inconsistent, that is, its first-order distribution and 
variance are independent of sample size, 7, smoothing is imperative 
when spectral factorization is employed and experience has shown that 
serious errors are obtained when unsmoothed estimates are used with 
the other autoregressive modeling techniques. Also, when the robust 
filter algorithm is used, the prediction residuals are measured on the scale 
of the estimated innovations variance. The accuracy of this estimate, 
and hence the reliability of the procedure, depends both on the actual 
stability of the pilot estimate and on what we estimate that stability to 
be. The latter effect enters in the form of a bias correction factor, a 
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function of the “equivalent degrees of freedom” of the pilot estimate, 
on the innovations variance estimate. With “short” data sets stability 
can only be obtained through liftering but even with long data sets where 
the Welch technique is applicable, liftering is used to improve the sen- 
sitivity of the stationarity test. 


3.2.1 The distribution of direct spectrum estimates 


Excluding the neighborhoods of the origin and the Nyquist frequen- 
cy, direct estimates of spectra are approximately distributed as a x2. For 
Gaussian data this result is exact and even in cases where the original 
data is reasonably nonnormal it is known (Fisher,*° Bartlett*1) to be a 
remarkably good approximation. Because the variance of such estimates 
is given by the square of their expected value, this fact emphasizes the 
need to start with a better estimate of spectra than the periodogram; 
estimates with low bias will have lower variance than estimates with high 
bias. 

The bivariate probability density function of direct spectrum esti- 
mates can be obtained from those given by Miller et al. 42 for Rayleigh 
processes 





(16) 


p(s1, $2) = e-(sits2)/(1-A) Jy (A) 
1-A 


1-A 
where both s, and sz have been standardized to unit level, I is the usual 
modified Bessel function, and A is the correlation between s; and sg given 
below by eq. (18). 

The characteristics of this distribution are most easily seen by con- 
sidering the conditional distribution p(s1|s2). For this distribution a 
critical point is given by sy = (1 — A)/A; at this point Op(s1|s2)/0s1|5,=0 
= 0 which for lower values of s2 resembles the univariate distribution 
and has its maximum at 0, while for larger values of sy the mode ap- 
proaches So. 

Figure 4 shows plots of the conditional distribution for so = 0.5 and 
1 and for values of A appropriate for the 47 prolate window at frequency 
spacings of 0.25/T, 0.5/T, 0.75/T, 1/T, and 2/T. 


3.2.2 Smoothing and frequency correlations of spectrum estimates 


There is a considerable literature on smoothing spectrum estimates 
(see for example Blackman and Tukey,!9 Parzen,3”-43 Papoulis,‘*) and 
the variance and distribution of smoothed estimates (Jones,2° Grenander 
et al.45) but much of this work is specialized to estimates based on the 
periodogram and cases where the different raw estimates included in 
the smoothing operation are uncorrelated. For the prolate data windows 
the latter assumption is unwarranted (as indeed it is even for the ex- 
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Fig. 4—Conditional probability density functions p(S1/S»2) for bivariate x3 distribution. 
Top set of curves, conditioning variable S. = 0.5; lower set, Sz = 1. 


tended periodogram) since, for the 4a window, the bias is only localized 
within a band of +4/T. In the more general case where the raw spectrum 
estimates are correlated, smoothing over a fixed bandwidth is less ef- 
fective and conventional smoothing techniques will be characterized by 
fewer “equivalent degrees of freedom” than given by the usual estimate. 
Most of the work on smoothing assumes that the true spectrum does not 
vary appreciably over the width of the smoother and under this ap- 
proximation the influence of smoothers on direct estimates is fairly 
simple to evaluate. 

To assess the effects of smoothing correlated spectrum estimates it 
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is necessary to examine their correlation properties. By expanding the 
fourth moment formula it can be shown that for Gaussian processes the 
covariance of the direct estimate at different frequencies is given by 


Cov{Sp(w + §), Sp(w -oj= 





~ f S(¢— ODE + w)D(E- wae 


1 * oe 2 
as f Se -abeE+ OBE- Hae] AD 





In this equation the first term is large only in the neighborhood of the 
origin (w = 0) while the second term is a convolution which, for ¢ = 0, 
equals E{Sp (w)}2. If, on the other hand, we set ¢ = A/2, the second term 
gives the covariance of estimates with a frequency separation of A in the 
vicinity of w. It is helpful to view the direct estimate, Sp (w), as a nons- 
tationary time series with a known covariance structure and in regions 
where the spectrum is locally white as a stationary series. As with other 
stationary series the second-order properties of the direct estimate in 
such regions are described by an autocorrelation function, which for unit 
spectrum is given by 

A(A) = |D*D*|? (18) 


Figure 5 shows the autocorrelation functions of the different direct 
spectrum estimates as a function of frequency separation, and again the 
local properties of the prolate tapers are striking by comparison to the 
very poor properties of the other estimates. It should be noted, however, 
that for the 47 window at the usual frequency mesh spacing of 1/2T the 
correlation between estimates is 0.9077 so that, as shown in Fig. 4, the 
distribution of estimates at this spacing is quite different than it is for 
independent estimates. 

It is frequently more convenient to work with the Fourier transform, 
=p, of this autocorrelation which we call the antespectrum of the esti- 
mator. Thus =p is defined by 


Zp(Q) = D*(Q)*D*(Q) (19) 


and is the spectrum of the spectrum estimate, Sp(w). The antespectrum 

is a function of quefrency, Q, which is a lag or time-like variable and its 

Fourier transform is the autocorrelation function, A, of the spectrum 

estimate expressed as a function of ordinary frequency separation. 
Defining a smoothed direct estimate Sp w(w) as 


5 if re 
Sp,wlwe) = 5 f Spo — NW) dé (20) 


in which the weight, W, is usually considered to be symmetric, positive, 
and integrating to 1. Since the spectral window of the direct estimate, 
Sp(w), is |D(w)|? the spectral window of the smoothed estimate is clearly 
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Fig. 5—Envelopes of autocorrelation functions of direct spectrum estimates. 


the convolution |D|2*W but the variance does not correspond to the 
usual interpretation of a spectral window in the literature on indirect 
estimates. Using the above definitions the influence of a smoothing 
operation, or lifter, may be described in the quefrency domain as a linear 
filter so that the antespectrum, 4p w, or the spectrum of the smoothed 
spectrum estimate is the product of the antespectrum, =p, and the 
power transfer function of the lifter. The variance of the smoothed 
spectrum estimate is the integral, over quefrency, of its antespectrum 
so that the estimate S p,w will have an approximately x? distribution 
with 


(een —1 
row= 2] fF W@PED(@<@ | (21) 
equivalent degrees of freedom. For direct estimates the antespectrum 
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Fig. 6—Antespectra corresponding to various data windows (the antespectrum is the 
spectrum of the spectrum estimate. 


is symmetric with a global maximum at zero quefrency. From eq. (21) 
it is clear that for liftering to be effective | W|? should be small when Zp 
is large. 

As indicated above the choice of weights is a complex subject which 
depends to a large extent on the intended application with perhaps the 
best linear smoothers obtained by modifying the technique described 
by Papoulis** to account for the data window. When this is done the 
Sturm-Liouville equation 


x 2 , = 
(D2(é)y’) + Ay = 0 (22) 


is obtained corresponding to his eq. (22) and can be solved by standard 
techniques. 

Figure 6 shows the antespectra corresponding to the various data 
windows. From these curves it is apparent that estimates using the 47 
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Fig. 7—Effectiveness of smoothing direct estimates. 


taper have more of their variance at low frequencies than the other es- 
timates. The bottom curve in this figure shows the antespectrum of a 
4x estimate smoothed with uniform weights over a bandwidth of £4/T 
which, when integrated, gives only 6.7 equivalent degrees of freedom. 
Figure 7 is a plot of the equivalent degrees of freedom resulting when 
direct estimates are smoothed. Two smoothers are used: simple moving 
averages which are useful for calibration purposes and the modified 
Parzen weights (Cleveland and Parzen*®). From these curves it is obvious 
that the correlation induced in the raw spectrum estimate by the data 
window can result in significantly fewer degrees of freedom than ex- 
pected on the basis of uncorrelated spectrum estimates. This effect is 
particularly noticeable with the 47 taper where, when bias considerations 
are excluded, the asymptotic efficiency is only 36 percent when only a 
single direct estimate is computed. As will be seen in the next section, 
the use of overlapped subsets results in variance efficiency. 
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3.3 The Welch technique and optimum subset spacings 


An alternative to smoothing across frequency is Welch’s method!! (see 
also Cooley et al.4”) in which the data is divided into several overlapping 
subsets, direct estimates computed on each subset, and the results 
combined. The individual subset estimates have the usual statistical 
properties of direct estimates but when used jointly one must also ac- 
count for the correlation between subsets. For the same reasons given 
in the previous section the effectiveness of this averaging must be ac- 
curately determined. Clearly spacing the subsets too close results in 
computational inefficiency while if they are spaced too far apart the 
procedure is statistically inefficient. : . 

Consider two direct estimates of spectra S; and Sz made on the do- 
mains (0, J) and (b, b + T) respectively. For Gaussian processes the 
covariance between these estimates is given by 


Cov{Sy(w), So(w)|b} = 





1 - og es ~ 2 
an J SWeBw- Bw + 9 ai| 


+|5- f sWe%Dw- pleas” (23) 





The first term of this expression is large near the frequency origin but 
elsewhere the second term dominates. Since D is an entire function it 
is clear that the covariance between the two estimates is governed pri- 
marily by the characteristics of the actual spectrum S, in the vicinity of 
w. In particular spectra having very narrow resonances or discontinuous 
characteristics will result in the subsets being correlated for large values 
of the offset b. The effect of this correlation is that averaging the different 
subset estimates does not give the usual reduction of variance so that 
the autoregressive model is unstable when only a few subsets are avail- 
able. When the correlation between subsets is low the distribution of the 
average of k subsets is nearly x3,. 

When the spectrum is locally smooth estimates of this type depend, 
in addition to the data window, on the two parameters T and b. The 
length of the individual subsets depends primarily on the fine structure 
of the process and will be discussed in Section V. The relative spacing 
of subsets, however, depends largely on the choice of the data window 
and in general there is an optimum spacing. Under the usual approxi- 
mation that the true spectrum is locally constant or linear and that we 
are interested in frequencies away from the origin, eq. (23) simplifies, 
and the correlation between subsets becomes the square of the equivalent 
lag window, Lp(b). 

As a measure of effectiveness of this procedure, assume that sufficient 
data is available to compute k subsets. Standardizing the local spectral 
level to 1, the variance of the averaged estimate is 


1790 THE BELL SYSTEM TECHNICAL JOURNAL, NOVEMBER 1977 


Vibe e251 +) LB(sb) (24) 
= tS -- s 
: k k s=1 ( k - 

We now consider the effect of adding sufficient new data to compute k 
+ 1 subsets and, by analogy with Fisher information, we measure the 
relative gain in information by 





eS eee eee 
ah) => | Cl a 


As k becomes large AJ, rapidly approaches the limit 


AI .(b) ==; —___ (26) 


[T/b] 
1+2 4 L3(sb) 
s=1 


This function is plotted in Fig. 8 for the different windows discussed 
earlier. When the subsets are spaced very closely relative to their length, 
no information is “‘missed” by falling between adjacent subsets, but on 
the other hand the subsets are highly correlated with each other so that 
the addition of a subset does not decrease the variance very much. For 
the 42 prolate window this situation remains true until the spacing be- 
tween subsets becomes about 0.25 to 0.30 of their length, after which the 
information recovery becomes rapidly less efficient. Because the com- 
putational burden rapidly increases as the offset is decreased, a subset 
spacing of about 0.29 of the subset length is used. For the less concen- 
trated windows this effect is less important. It should also be noted that 
the higher information recovery of the 47 prolate window evident here 
is consistent with the fact that it has a broader frequency response, so 
that, apart from bias considerations, the overall efficiencies of the 
techniques are similar. When bias considerations are included the effi- 
ciency of the prolate window is much higher. 


IV. DATA PREPARATION 


Assuming that aliasing and noise effects have been properly kept at 
a minimum there are usually two steps of data preparation necessary 
in time series work. The first is the elimination of gross errors and the 
second is the removal of deterministic mean value functions. 

Gross errors are inevitable in very large data sets, see Hampel,*® and 
experience has shown that the WT4 project is no exception to this 
rule. 

Errors which are large and easily visible are best removed at an early 
stage in the processing. A simple strategy which works for both large 
errors and missing values is as follows: 


(t) Data points in serious error are tagged, either on the basis of vi- 
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Fig. 8—Asymptotic relative information gain as a function of subset base offset. 


sual examination or automatically on the basis of built in validity 
checks.* 

(ii) Predictors and interpolators were generated either from untagged 
data or, if similar data sets were available, from the autocorrelations of 
these sets. 

(tit) When the error is isolated it is replaced by interpolation. If the 
errors could not be considered isolated those adjacent to the longest 
stretch of good data were corrected first by prediction from the good 
section. After all the tagged points had been initially corrected by pre- 
diction, the corrections were recomputed by interpolation using the 
initial correction as a basis for the interpolations. 


t Records of long range mouse data were coded to provide indications of hardware 
malfunction. 
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However, not all errors in time series data are obvious from a plot. In 
situations where the spectrum covers many decades an error may be 
insignificant on the scale of the process variance but catastrophic 
compared to the innovations variance. At the data preparation stage 
these errors are neither easily detectable nor troublesome and so their 
correction is deferred to the prewhitening part of the process where they 
are effectively eliminated by the robust filtering process. 

The other stage of data preparation is the removal of deterministic 
mean value functions from the data. A simple example of a series with 
a nonconstant mean value function is given by the axis curvature of a 
waveguide following a planned route bend. 


The usual approach in time series analysis with problems of this type 
is to remove the “trend” using orthogonal polynomial regression tech- 
niques. This approach has proved unsatisfactory primarily because such 
a high-degree polynomial is required to approximate the mean value 
function that the residuals bear little resemblance to the stochastic part 
of the process. 


A method of removing trends in data which has proved generally ef- 
fective is based on the use of polynomial B-splines. A B-spline of order 
k is a piecewise continuous polynomial of degree k — 1 defined by an 
array of knots, some of which may be multiple. The continuity properties 
of these functions are controlled by the knots; the spline is discontinuous 
at a knot of multiplicity k, has a discontinuous derivative at knots of 
multiplicity k — 1, and so on. At simple knots, or knots of multiplicity 
1, the spline has k — 2 continuous derivatives. Details of the theory of 
B-splines are contained in a paper by Curry and Schoenberg,*? a recent 
paper by de Boor®® describes computational aspects, and Horowitz*! 
discusses the characteristics of splines with equispaced simple knots in 
terms of their frequency domain characteristics. 


Figure 9 shows a plot of the measured elevation of a waveguide line 
and an approximate mean value function generated through the use of 
B-splines. By choosing a spline with few knots, indicated on the figure, 
a simple fit to the gross topology of the run is obtained so that the 
“roughness” of the installation is readily apparent. 

A second example of the use of polynomial spline mean value functions 
is shown in Fig. 10, which is a plot of the vertical output from a mea- 
surement of axis curvature on a waveguide tube supported on Airy point 
supports (see Fox et al.52). In this case most of the indicated curvature 
is a result of the tube sagging under its own weight and this effect is 
readily calculable and is shown by the dashed line. As a check that this 
removal is not distorting the spectrum of the actual distortions in the 
tube the ratio, an F statistic, of the average of 10 estimates of the spec- 
trum of the detrended vertical curvature to the spectrum of the hori- 
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Fig. 9—Example of a B-spline fit to waveguide elevation knots. The spline is order 4 
with knots of multiplicity 4 at —0.1 and 90 meters, simple knots at 23.5 and 56.5 me- 
ters. 


zontal curvature was computed. No significant differences could be 
detected between the two sets of spectra. 


V. PILOT SPECTRUM ESTIMATE 


The actual process used to generate the pilot spectrum estimate is a 
combination of the two smoothing approaches described in Section III. 
The use of subsets allows the test for stationarity and, because this test 
is more sensitive when applied to smoothed data, a logical step is to 
smooth the subset estimates individually. However for the stationarity 
test to be effective it is necessary that the different subset estimates be 
essentially uncorrelated at any given frequency. This requirement results 
in the base offset between adjacent subsets being more than about 57 
percent of the subset length, which is larger than is desirable for the most 
effective use of the data from an information recovery viewpoint. The 
obvious solution is to compute the subsets with the 29 percent offset 
mentioned above and use every other subset in the stationarity test. 

A further advantage of the use of subsets is that a significant im- 
provement in the accuracy of the pilot estimate can often be obtained 
by combining the different subset estimates in a robust manner instead 
of by the usual arithmetic average. Denoting the ordered subset estimates 
by S;(w) with Si(o) < S5(w) < -- < &.(w), a robust estimate Sw) may 
be formed as 


k’ 
S() = z 0S; (w) (27) 
i= 
where the weights, {0;}, which depend on k’, are chosen so that Sisa 
minimum variance unbiased estimate of S. General techniques for 


forming such estimates are given in Lloyd®? and the specific means and 
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Fig. 10—Example of a spline mean value function. Data represent vertical curvature 
output from a rotating-head mouse. 


covariances of the order statistics for gamma distributions required are 
given in Sarhan and Greenberg and Prescott.®° 

For the unsmoothed subset estimates the means and covariances are 
particularly simple and the weights are given explicitly by 

0; = = J<k’ (28) 
R+1—k’ 

k’ 

and the variance of S() = E{S (w)}2/k’ so that the efficiency, relative to 
an uncensored estimate, is just k’/k. It is shown by Mehrata and Nanda®® 
that this estimate is maximum likelihood. This procedure is most ef- 
fective for eliminating the effects of the occasional gross outlier missed 
in the data preparation stage but, unlike the robust filter algorithm, is 
ineffective against numerous small outliers. 

As mentioned earlier, the length of the individual subsets is dependent 
on the fine structure of the spectrum to be estimated. A simple method 
of estimating this length (which within fairly broad bounds is not critical 
since the final estimate is primarily responsible for fine structure) is to 
compute a moving average representation of the process. For this pur- 
pose the Wiener canonical spectral factorization approach is ideally 
suited and, if in eq. (42) below, the sign of the summation is reversed and 
the expression Fourier-transformed, a moving average representation! 


0,’ = 


!This moving average representation is the minimum delay causal nonrecursive 
(transverse) filter generating the observed process from white noise. The convolution of 
the moving average with itself gives the autocovariance function of the process. 
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Fig. 11—Canonical moving average representation of vertical curvature gauge output. 
Representation based on the average of 102 450-meter subsets. 


is obtained instead of an autoregression. Figure 11 shows such a model 
for the vertical gauge output for the Netcong field trial data and it is 
apparent that most of the weight is concentrated within a 60-meter 
range." T’o allow for the heavy tapering effect of the 4x prolate window 
a subset length of 160 meters was used. 


Vi. CONSTRUCTION OF AUTOREGRESSIVE MODELS 


The basic reason for computing a pilot spectrum estimate is to permit 
the design of an accurate prewhitening filter and the subsequent use of 


+ The discontinuities visible in this plot near 9 and 18 meters are due to the couplings 
but due to the randomized guide lengths this effect is rapidly suppressed with increasing 
separation. 
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a high-resolution spectrum estimation technique on the filtered data. 
Prewhitening filters are subject to several constraints; for example their 
transfer function must have no zeroes in the frequency range of interest 
[see eq. (61)], their design must be readily automated, they must have 
finite impulse response, they must be numerically well conditioned, and 
they must be absolutely stable. The prediction error filter satisfies these 
requirements. 

Moreover, since the prediction error filter is causal and depends on 
the canonical autoregressive representation of a stationary time series 
it has the further major advantage that it may be readily robustified as 
described in Section VII. The use of the prediction error filter is therefore 
conditioned on one’s ability to estimate the parameters of an autoreg- 
ressive model of the process and this problem is the subject of the present 
section. Current work on autoregressive modeling uses two distinct ap- 
proaches; direct solution of the Yule-Walker equations such as described 
by Pagano,®*” Ulrych and Bishop,°? Makhoul,®? and spectral factorization 
as described in Bhansali.®°.6! Following a brief review of these two ap- 
proaches a composite technique is described which exploits features of 
both. The section concludes by considering three alternative methods 
of computing prewhitening filters. 

In the autoregressive representation of a discrete time process the 
value, x;, of the series at time t is given by the sum of a regression on the 
past values of the series and an independent random component, &,, 


Dp 
xe= & + Xu OjXt¢—j (29) 
i= 


An equivalent description is to regard the regression on the past of the 
series as a prediction of the value of the process at time ¢ so that the 
random component, &;, represents the “new” information or innovations 
of the process. The length of the predictor or order of the regression is 
denoted by p which may be infinite. Such processes and questions related 
to them are discussed extensively in the literature; see for example 
Hannan,®? Koopmans,”4 Box and Jenkins,®° or Doob.28 The papers by 
Kailath,®4 Kailath and Frost® are also relevant to these problems. 


6.1 Yule-Walker equations and the Levinson algorithm 


The basic equations determining the autoregressive coefficients are 
derived by minimizing the one-step prediction variance 


jE ((o- Zorn] 


with respect to a; for j = 1, 2p and are known as the Yule- Walker 
equations 
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D 
Pr = 2a a!) py;-p) k=1,2,---p (31) 
j= 


In these equations p; is the autocorrelation function of the {x} process 
at lag k. These equations are not only linear in the a’s but also Toeplitz, 
so that the matrix elements depend only on their distance from the main 
diagonal and for real series the p X p matrix has only p distinct elements. 
Since these equations are linear, they may be solved using standard 
techniques such as the QR algorithm (see Dahlquist et al.6*). However, 
because of their special structure, special procedures are available for 
their solution which require only p? operations instead of the p? required 
with general linear equation techniques. Also, because fewer operations 
are required, roundoff errors are reduced and the faster algorithms can 
be more accurate. 

Generally these fast algorithms are similar in structure to the recursive 
solution of the Yule-Walker equations discovered by Levinson.®” One 
convenient and numerically stable variant is due to Durbin,® which in 
the notation of Ramsey,” is initiated using 


1 = af? = py (32) 
of = 1- oj (33) 
and continued for k = 1, 2, -, p — 1 by 
bre = ahhh) = [pass =o af oxs-5| i oF, (34) 
ae) = a®) = brrieey; PSA Qees he (35) 
Oh+1 = oR(L — bis) (36) 


In these equations the a{")’s are the autoregressive or prediction coef- 
ficients for k step prediction, the ¢@ sequence is known as the partial 
autocorrelation function, and a? is the k step relative prediction error. 
In the original Levinson algorithm the expansion 


Rk 
oh=1- > ally (37) 
J= 


obtained by substituting eq. (31) into (30) was used in place of eq. (36). 
Analytically these equations are identical but the latter is both slower 
and also has much poorer numerical properties than Durbin’s form. 


6.2 Spectral factorization 


One drawback of the Toeplitz matrix formulation is that it does not 
provide much insight into the actual minimization process and it is 
helpful to rewrite the equations in terms of a prediction error filter where 
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we define 


af) =—-1 (38) 
and the negative error sequence 
Zi. =O x) (39) 
= > afP x; 


Note that the {z} sequence is a result of a linear causal convolution, or 
filtering operation, applied to the {x} sequence. The transfer function 
of this filter is 


A®)(w) = 3 afPe—iok (40) 
k=0 
so that the spectrum of {z} is S,(w)|A)(w)|? with the corresponding 
variance 


a2 feces 
oh = f- "Sx (w)|AP(e)|2 de (41) 


where S,.(w) is the spectrum of the {x} process. As p ~ © the spectrum 
of the error sequence approaches a constant so that the problem is to 
choose the causal filter in such a way that |A|? is small whenever S, is 
large. For A a trigonometric polynomial of degree p the problem has been 
completely solved by Szegé7! and the recursion formulae for the or- 
thogonal polynomials obtained are essentially similar to those above. 
An alternative solution is provided by Wiener’s® canonical spectral 
factorization where the filter transfer function A is represented as 


A(w) = —exp le > cyeiet| (42) 


so that the variance of {z} is given by 
c= i f. S,.(w)e—22 k= 1cxcoswkdw (43) 
20 =_ 


Direct minimization of this expression as a function of the cz’s is im- 
practical due to the complexity of the resulting equations. Wiener’s 
approach is to identify the c, with the Fourier series coefficients of In S,, 
that is 


1 T 
ch = — f. cos wk In {S,(w)} dw (44) 
20 —7 
The sequence c,, is referred to as the cepstrum. 
It is important to notice that the series in eq. (42) does not include a 


Co term because the constraint imposed by eq. (38) implies that co defines 
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the minimum. This is most easily seen by substituting the Fourier series 
representation (44) into eq. (41) with the result 


Pate E fin 8.00) da | (45) 


oj is the innovations variance of the process. For this procedure to be 
valid the spectral distribution function must be absolutely continuous 
which implies that autoregressive representations are invalid for process 
containing periodic components. The procedure is formal and as men- 
tioned in Wiener and Masani,’? the sense in which it converges is un- 
known. 

D. Preston”? has observed that in practice these convergence problems 
may be avoided by ieee the formula ee 


AG) =n — op [-< = a in sy ent ue <= an| (46) 
pol GO; 

inside it’s radius of convergence, ie for u < 1, rather eas on the radius 

of convergence as does the Wiener approach. With this modification one 

obtains 


A, (w) = —exp I De canter ieh | (47) 


so that the coefficients a; may be computed by Fourier transforming 
A, (w) and dividing by w/. 

These two techniques have been described as if the actual spectrum 
were known. When applied to an estimate of the spectrum, things are 
more complex and neither technique has a clear advantage over the 
other. The disadvantage of the first approach is that it works explicitly 
with the autocorrelation function. The range of spectra common in 
waveguide work is so large that use of the autocorrelation function is 
numerically undesirable in that information corresponding to the lower 
parts of the spectrum may be lost due to numerical roundoff errors. The 
second method is numerically stable but produces a filter with a very 
long impulse response which reproduces all the details of the spectrum 
on which it is based, including those due to sampling. Since the robust 
filter algorithm works in the time domain the shortest autoregressive 
model which retains the statistically significant features of the spectrum 
is desirable. 

Cleveland”® and Bartholomew” have described several sources of error 
in prediction problems. Of these the most critical appears to be a result 
of sampling variability in the spectrum estimate. As an example consider 
the estimate of innovations variance, é7 obtained by using the pilot 
spectrum estimate, S, in place of the spectrum, S,, in eq. (45). This es- 
timate is described by Davis and Jones”” except that their bias correction 
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is not used in steps involving the model formulation. (‘The bias correction 
is used to set the scale of the residuals for the robust filter algorithm.) 
Grenander and Rosenblatt}® give a formula for prediction error in the 
case when the predictor is based on an estimate of spectrum, S, rather 
than on the true spectrum, S, of the process 


a2 2 1 ¢7,, Ss) 1 ¢7S() 

Pe fee = J. ” S(e) do| Qn _ co 
Periodogram estimates are distributed as x3 so that E{S —1(w)} = 0, While 
this result is based on the Wiener spectral factorization method and so 
applies to prediction using the entire past, it appears to give a good in- 
dication of the behavior of autoregressive fits even for relatively compact 
predictors. Further information on the effects of smoothing on the es- 
timated innovations variance is available in Jones.”8 


6.3 Reduced factorization 


A method which exploits many of the advantages of both of the pre- 
ceding approaches without having the fatal flaws of either is to reduce 
the result of spectral factorization. In the reduced factorization approach 
one begins by creating a long autoregressive model using Wiener’s 
spectral factorization method described above and then converts it to 
ashorter representation using the Levinson recursion formulae. In this 
reduction the key equation is (35) which, by combining the updates for 
a!” and a{");_;, may be written backwards. When written for use in a 
downwards recursion this formula becomes 


(k+1) k+1).. 
Qa + Aba; 
op") es Pr+1 ar (49) 


1 — dhe 
Similarily the k-step prediction variance, a? may be obtained from 074, 
by using eq. (86) backwards starting from the estimate of innovations 
variance given by eq. (45). 

The major disadvantage of the reduced factorization technique is that 
it is somewhat slower than either of the standard techniques individually. 
Of these the Levinson recursion is the faster: it requires only a single 
Fourier transform to convert the pilot spectrum to a sample autocorre- 
lation function and then p? operations for the actual solution. In practice 
it is necessary to “search” for the correct order of the autoregression. 
Since this search is never carried past Dmax = V T the total computation 
time is ~T' In T. Since spectral factorization requires three Fourier 
transform operations, its speed is comparable with that of the Levinson 
technique. Reduced factorization requires an additional T? operations 
and is therefore considerably slower when very large data subsets are 
being used. 
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As mentioned earlier the most serious flaw with the Levinson approach 
is a result of roundoff errors in the Fourier transform used to convert 
the pilot spectrum estimate to autocovariances and is only serious when 
the range of the pilot spectrum estimate is large. Roundoff characteristics 
of fast Fourier transform algorithms are well understood (see Kaneko 
and Liu’’) and consequently the characteristics of the pilot spectrum 
estimate relative to the computer precision may be used to select the 
“best” procedure: when the range of the pilot estimate is low the Lev- 
inson-Durbin algorithm is used, but in cases where the range is large 
reduced factorization is preferred. 

With either approach the order of the autoregressive representation, 
p, has been chosen as the value of 7 for which Parzen’s!® criterion 


ue 
T 


attains its minimum. Within reasonable bounds the actual order selected 
is not critical as the autoregressive model is used as a prewhitening filter 
and not as a spectrum estimate. (The function ¢2/|A)(w)|? is known 
as an autoregressive spectrum estimate. See Akaike,®° Gersch and 
Sharpe.®!) Berk®? gives conditions on the order, p, for obtaining a con- 
sistent model of the process. 

The actual method used to determine the prediction error filter is a 
combination of the two methods discussed in Sections 6.2 and 6.3 as 
shown in the flow diagram, Fig. 12. In its general form this spectrum 
estimation technique is an iterative process and intermediate estimates 
are used to update the pilot estimate of spectrum and the prediction error 
filter. In cases when iteration is used it is stopped when the estimated 
innovations variance stabilizes. 


ie) 
P(r) =1-44 (50) 
Oo; 


6.4 Alternatives 


Since the sequences of steps which is being used here to generate an 
autoregressive model is by no means obvious it is worthwhile to briefly 
examine the alternatives. The obvious technique of eliminating the pilot 
spectrum estimation and transformation to autocorrelations procedure 
and estimating the sample autocorrelations directly is not done because 
of the high bias, discussed in Section IV, of this estimate. 

The second possibility is to form the pilot estimate of spectra and then 
design a conventional digital filter for the prewhitening operation. De- 
tails of this approach using Gegenbauer filters are given in Thomson.®% 
The drawback is that such filters are incompatible with the robust filter 
algorithm. 

A third alternative is to directly estimate the partial correlations using 
Burg’s® algorithm. In this approach an autoregressive model is estimated 
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Fig. 12—Iterative estimation procedure. 


by minimizing the sum of the forward and reverse prediction errors 


ere) = (n -> alP'inse) + s (an -> Pint) (51) 


n=1 k=1 n=pt1 k=1 
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successively for p = 1, 2--- under the constraint that the covariance matrix 
is Toeplitz so that the autoregressive coefficients, a{?’, are updated using 
eq. (35). This method, also referred to as the “maximum entropy” ap- 
proach, gives the partial autocorrelations directly and constrains them 
to be less than 1 in magnitude. 

Limited Monte-Carlo studies indicate that spectra based on autore- 
gressive representations obtained in this way have exceptionally high 
variance whenever the order, p, of the autoregression is carried far 
enough to reveal details of the spectrum. Other problems with the Burg 
algorithm as a spectrum analysis technique are described in Chen and 
Stegun.® As a prewhitening algorithm it has been used on individual 
tubes with reasonable success. These “partial Burg” routines have been 
effective in situations where a low order autoregressive representation 
is adequate for the prewhitening filter, the range of the spectrum is low, 
and very compact code is required. 


Vil. ROBUST FILTERING AND PREWHITENING 


One of the most useful data analysis tools developed during the course 
of this work is the robust filter algorithm. This is a nonlinear technique 
designed to eliminate the effects of occasional “outliers” in the data from 
the final spectrum estimate, where, as mentioned earlier, outliers are 
measured on the scale of the innovations process. As an example of the 
magnitude of this problem, the typical output of a tubing curvature 
gauge is about 15 microns rms whereas the scale of the innovations 
process is about 0.5 microns. This is considerably smaller than the size 
of typical dust particles and, since this gauge operates primarily in a 
tubing mill, the probability of some dust particles being measured is 
quite high and the need for robust filtering is evident. 

The robust filter algorithm differs from linear filtering in that most 
of the data passes through the “filter” without modification and only 
those points which are basically unpredictable from past values of the 
series are changed. The characteristics of the filtering algorithm are 
controlled by providing (i) an autoregressive model of the process, (it) 
an estimate of the innovations variance, and (iii) an influence function. 
In practice the autoregressive model and innovations variance must be 
estimated from the data and it has been found that the algorithm works 
well even with surprisingly inacurate models. Further details and ex- 
amples on this procedure are available in Kleiner et al.8° The steps of 
this procedure, as it is currently implemented, are listed below. Section 
7.2 summarizes results (see Kleiner et al.®”" for details) relevant to the 
choice of influence function, and in Section 7.3 an example of the action 
on contaminated data is given. Further information on robust procedures 
is available in Huber’? and Hampel.*® 
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7.1 The robust filter algorithm 


We assume that the observations, {y}, consist of the process of interest, 
{x}, plus occasional outliers, {v} 


VR = Xp + VR (52) 


Based on this contaminated data the robust filter algorithm produces 
an estimate, {<}, of the “core” process by the following steps: 


(t) A prediction, %,, is made from the filtered sequence using the 
autoregressive coefficients obtained by the methods discussed in Section 


VI. 


En = = al) in—p (53) 
k=1 
(it) A weight is defined which depends on the difference, y, — Xn, 
between the actual observation, y,, and the prediction. This difference 
is normalized by the scale of the innovations process, cp. (The scale is 
the square root of the prediction variance estimate, Gs, with the bias 
correction given in Davis and Jones.””) 


Wn = W e) (54) 
Sp 

In the applications described here W is an even function with W(0) = 
1 and W(~) = 0. When multiple errors are encountered the scale, cp, 
used in this formula is replaced by an approximation of the k-step pre- 
diction variance. 

(iit) The output of the robust filter algorithm is an estimate of the 
core process, <,, formed by the weighted average of observation and 
prediction 


En = WnYn + (1 _ Wn)kn (55) 


The effect of this procedure is to leave the data unmodified where the 
prediction errors are small and to replace the data with its prediction 
at points where the prediction errors are gross. The action taken when 
the prediction errors are near the expected extreme for the given sample 
size depends on the weight function which will be discussed below. In 
spectrum estimation applications the desired output is usually not the 
filtered sequence but rather the prewhitening residuals 


Zn Sa oe ai (56) 


The zn may be described in terms of an influence function (see Ham- 
pel®9) 


y(e) = eW(e) (57) 
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applied to the relative prediction error, (yp; — %,)/op, but the notation 
is deceptive in that it deemphasizes the fact that the weighting procedure 
also influences the prediction for subsequent steps. 


To use this algorithm the filtered sequence must be initialized on the 
first p points. When the data is only slightly contaminated the raw data 
has been used to start the process but when the contamination is more 
severe special precautions must be taken. 

The detailed behavior of the algorithm depends on the choice of weight 
function, and this represents a compromise between rejecting valid 
outliers of the innovations and accepting the occasional erroneous data 
point. Considerable information is available on the choice and charac- 
teristics of influence functions for robust estimates of location (see An- 
drews et al.®°), but this information is of limited utility in time series 
applications since in location estimates there is no concern with fre- 
quency response characteristics. It must be remembered that this op- 
eration is nonlinear and that nonlinear operations on time series gen- 
erally change the spectrum in complex ways. Because of this the weight 
or influence function must be chosen in such a way that the spectral 
content due to the induced nonlinearities is much less than that due to 
the presence of errors in the data. 

Several different weights have been used. Of these the best found to 
date is a result of motivation by the extreme value distribution for dis- 
tributions of exponential type (see Kendall and Stuart®!) and is defined 
by 


W(u) = exp {—evollul—u0)} (58) 


in which 
1 
= -1 =—_— 


® being the normal cumulative distribution function and N the sample 
size. This influence function, shown in Fig. 13 for N = 1000, is very linear 
in the center and, at about +3<, decreases rapidly to zero. 


7.2 Spectral distortions resulting from robust filtering 


In its most general form, use of the robust filter algorithm is alternated 
with the model formation process as shown in Fig. 12. In this iterative 
mode the output from the filter is used to generate a better autoregressive 
model which is used to filter the data and so on. This kind of iterative 
procedure has been used for some difficult data sets and was found to 
converge to a stable estimate of spectrum very rapidly. Typically two 
or three iterations are required on short series (for example, some dis- 
tortions in individual tubes) where the range of the spectrum is very large 
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INFLUENCE FUNCTION, Woy (x) 





—5 —4 -3 —2 —1 0 1 2 3 4 5 
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Fig. 13—Extreme value influence function. 


and the outliers are small relative to the scale of the process but large 
compared to the scale of the innovations. With very large data sets, such’ 
as those from complete mode filter sections of the field trial (which av- 
erage 80,000 data points), a single iteration has been used and found 
satisfactory. 

If one assumes that this iterative process has converged, it is possible 
to describe the distortions introduced into the spectral density estimate. 
At convergence the autoregressive parameters, a, describe the estimated 
process, {%}, and are solutions of the Yule-Walker equations based on 
the estimated process. Then by computing the expectation of ¢,_, with 
£, using the representation eq. (55) for the latter, it is found that the &;’s 
also are solutions of a set of robust Yule-Walker equations 


P A A 
Yn — a AjXn-j 
E £n—-2W | ——_ =0 kR=1,+++,p (60) 


op 
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An alternative viewpoint is to regard the algorithm as the solution of 
minimizing a nonquadratic loss function of y,, — X, with respect to the 
a;,’s and it can be shown that the solution to this problem also yields the 
robust Yule-Walker equations provided that the influence function is 
such that the error sequences, 0£,,/04; are small. This is a reasonable 
requirement: small changes in the process specification should not result 
in large changes in the filter output. The satisfaction of this condition 
depends on the choice of influence function, ¥. It can be shown that the 
scale of the error'sequences depends on 1 — y so that influence functions 
having very high curvature in regions where the probability density 
function of the innovations process is large result in larger errors than 
influence functions which are more linear in such regions. The most 
important property of the algorithm, however, is that, for reasonable 
influence functions, the effect of the nonlinearities on the spectrum es- 
timate, is proportional to the spectrum so that the net effect is a slight 
downwards bias. The scale of the bias factor is E{éy(£)} and, for the 
dominant error terms, is independent of frequency. 


7.3 Action of the robust filter on contaminated data 


The intent of the robust filter algorithm is to reduce the effects of 
outliers and erroneous data from the final estimate of spectra. Since the 
choice of influence function is to some extent distribution dependent, 
it is also of interest to observe the effect of this algorithm in a direct 
manner. It is also interesting to check to what extent a normal assump- 
tion on the basic data is warranted. Since the high serial correlations 
existing in most time series in the physical sciences make the usual tests 
for goodness-of-fit to a given distribution inapplicable this must be done 
cautiously. A very conservative approach is to find some lag, 79, such that 
the autocorrelations at multiples of this lag are small and test samples 
taken at this spacing for normality. Since the spacing required to obtain 
uncorrelated data may be large, this approach is rather inefficient. An 
alternative is to consider the residuals from the prewhitening operation. 
Since these residuals are generally very small, usually only a few times 
the quantization level, this method is very sensitive to outliers and 
measurement errors. Figure 14 shows a Q-Q plot of the residuals from 
a linear prewhitening operation and it is clear that the apparent distri- 
bution has very heavy tails. If the actual residuals are plotted as a time 
series, Fig. 15, it is clear that at least part of the long-tailed characteristics 
are a simple consequence of the fact that in linear prewhitening each 
outlier in the original series is converted into p + 1 outliers in the residual 
series. In Fig. 16 a Q-Q plot of the residuals from the robust prewhitening 
algorithm is given and the contrast is striking. In this case the residuals 
are quite close to normal and in agreement with tests made on other 
sections of the line. 
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ORDERED AND STANDARDIZED PREDICTION RESIDUALS 
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Fig. 14—WT4 field evaluation test horizontal curvature gauge output. Residuals from 
a linear prediction error filter. 


Vill. FINAL ESTIMATE OF SPECTRUM 


Prewhitening converts the data from a highly correlated into an almost 
uncorrelated form whose spectrum has a low dynamic range. Estimates 
of such spectra are best made with windows which have high frequency 
resolution and do not need the extreme sidelobe suppression used for 
the pilot estimate and the Tukey spliced cosine window has been used 
for most such applications. 

The final estimate is intended primarily to extract details of the 
process: consequently the data is not split into subsets and the estimate 
is not smoothed by liftering. In cases where “smoothing” is done it has 
been by the nonlinear methods discussed in Section 2.7. These tech- 
niques might be described as “inverse influence” in that individual points 
are lumped into a moving average except when they are outliers in which 
case they are used instead of the average. This procedure is a useful aid 
for spotting peaks and other low level features in the spectrum. 
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PREDICTION ERROR 
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Fig. 15—WT4 field evaluation test horizontal curvature gauge output. Residuals from 
a linear prediction error filter. 


In the final estimate of spectrum it is necessary to correct for the 
prewhitening operation so that the result is expressed as the ratio 


Spee (61) 


. 2 
2 Ape —iwk 
k=0 








in which S,(w) is a direct estimate of the spectrum of the prewhitened 
residuals [eq. (56)], and the denominator is the power transfer function 
of the prediction error filter defined in eq. (39). 
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ORDERED AND STANDARDIZED PREDICTION RESIDUALS 
ro) 
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Fig. 16—WwT4 field evaluation trial curvature gauge output. Prediction residuals from 
the robust filter algorithm. 


The validity of this form depends on the assumption of the indepen- 
dence of the filter and the residuals and is discussed briefly in Grenander 
and Rosenblatt.18 With the prediction error filter this is a reasonable 
assumption in that the filter depends on the partial autocorrelation 
functions up to lag p while any residual structure is primarily the con- 
tribution of the partial correlations for higher lags. This assumption is 
also supported by Whittle’s!© observation that the information matrix 
splits into one part describing the structure of the process and a second 
part describing the innovations sequence. 


IV. CONCLUSIONS OF PART | 


A technique for estimating the power spectral density function of a 
stationary time series has been described which is robust, accurate, and 
computationally straightforward. Part II of this paper will give examples 
of its use and comparisons with standard techniques. 
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APPENDIX A 


Formulae for prolate spheroidal data windows 


A convenient expansion of the prolate spheroidal wave function data 
windows is given in Flammer?® Section 3.2. This expansion is a power 
series in terms of 


U=(1-x)(1+x) 


Computationally it is advantageous to rewrite the power series using 
Horner’s rule and for c = 47 the expansion is: 


= 2Ax 
Dool4x, x) = Ve CEC ECECECECKCKC CR KEK 


+2.6197747176990866d — 11 U + 2.9812025862125737d — 10) U 
+3.0793023552299688d — 09) U + 2.8727486379692354d — 08) U 
+2.4073904863499725d — 07) U + 1.80113859410323110d — 06) U 
+1.1948784162527709d — 05) U + 6.9746276641509466d — 05) U 
+3.5507361197109845d — 04) U + 1.5607376779150113d — 03) U 
+5.8542015072142441d — 03) U + 1.8482388295519675d — 02) U 
+4.8315671140720506d — 02) U + 1.0252816895203814d — 01) U 
+1.7233583271499150d — 01) U + 2.2242525852102708d — 01) U 
+2.1163435697968192d — 01) U + 1.4041394473085307d — 01) U 
+5.9923940532892353d — 02) U + 1.4476509897632850d — 02) U 
+1.5672417352380246d — 03) U + 4.2904633140034110d — 05) 
The expansion for the higher resolution window with c = 7m is: 


Doo(m, x) = a mC 


+5.3476939016920851d — 11 U + 2.2654256220146656d — 09) U 
+7.8075102004229667d — 08) U + 2.1373409644281953d — 06) U 
+4.5094847544714943d — 05) U + 7.0498957221483167d — 04) U 
+7.7412693304064753d — 03) U + 5.5280627452077586d — 02) U . 
+2.2753754228751827d — 01) U + 4.3433904277546202d — 01) U 
+2.2902051859068017d — 01) 


In the forms given here both functions have been normalized for use as 
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data windows. In this application x takes on values 





2t—1 
xt = T = 15 a er 
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